macOS Upgrade Bug

We attempted to upgrade a remote Mac running Mojave and ran into the macOS upgrade bug. This post walks through the steps we took to remotely recover the Mac and finish the upgrade.

tl;dr

Context

This was a 13″ Intel MacBook Pro that shipped with either High Sierra or Mojave. After the install failed, there was an attempt to reinstall macOS using internet recovery. The recovery environment got upgraded to Big Sur or maybe Monterey.

Between these two failed installs, the user accounts got messed up. I later find out that there was no longer an admin account on the Mac.

After the two failed upgrade attempts, when booting into either local recovery or internet recovery we were presented with a macOS Recovery screen requesting a recovery key. We’ve run into a similar state before where macOS would boot into FileVault recovery and request a recovery key. Typically an NVRAM reset actually resolves this (shocking, right?) In this case, it didn’t work (less shocking).

My theory now is that there weren’t any admin accounts at this point for the recovery environment to authenticate access to the sealed volume. The only presented option at this point was Erase Mac.

Thankfully this being an older laptop, I thought of one more thing to try: the ‘legacy recovery environment‘. I asked the user to shutdown their Mac, and then had them boot while holding down the Option-Shift-Command-R keys. Thankfully this loaded an older internet recovery. From here I was able to walk the user through the remediation steps.

Some Advice

After walking an end-user through the remediation steps I’ve got some advice that I’d like to pass along before the remediation steps.

  • Instead of running the commands with the full path, cd into the base path (hat tip to Eric for the suggestion). This way there is less of a risk for anyone to accidentally delete all of their data. This will also help test if a user is typing the commands out correctly.
  • Use the find version of the commands instead of the rm version. There’s less of a chance for things to go wrong if the commands are typed incorrectly.
  • Have them type the command out, and then send you a picture of the typed out command before they hit enter. This will allow you to review and correct any mistakes before they execute the command.

Remediation Steps

  1. Shut the computer down
  2. Boot into legacy internet recovery by holding down the left-side Option-Shift-Command-R keys and pressing the power button
  3. Select your language of choice
  4. Open Disk Utility
  5. Select ‘Macintosh HD – Data’ and click Mount
  6. Quit Disk Utility (Disk Utility -> Quit)
  7. In the menu bar, open Utilities -> Terminal
  8. Run the following commands — Note: each command may take a while to run. You’ll know it’s complete when the prompt appears to accept commands again and you see the bash-$ prompt.
  9. Remember: have the user send a picture of each typed out command for you to validate before they run it
  10. cd /Volumes/Macintosh\ HD\ -\ Data/Previous\ System/private/var/
  11. find folders/*/*/C/com.apple.mdworker.bundle -mindepth 1 -delete
  12. find folders/*/*/C/com.apple.metadata.mdworker -mindepth 1 -delete
  13. Once both commands are finished, reboot — click the Apple logo -> Restart (or just type reboot while still in Terminal)

The install should finish now, hopefully.

Post-Upgrade

After the upgrade was finished, I ran into one last problem. I hopped into the computer and attempted to authenticate as an admin, but it wasn’t working. Eventually I realized that the Mac no longer had any admin accounts.

In the end, I was able to resolve this using ConnectWise Control which lets us run commands on remote Macs as root. Running dseditgroup -o edit -a "$USERNAME" -t user admin resolved this for us.

Eric and I were talking about this and I wanted to offer up a few suggestions that we came up with in case anyone else runs into this issue.

If you have a tool like JAMF or Munki that users can self-service install software, they could

  • Install Privileges and escalate their user to an admin and then permanently upgrade a user to an admin user
  • Install a payload-free package that runs the dseditgroup -o edit -a "$USERNAME" -t user admin command to upgrade a user to admin

Also, your MDM vendor might have the ability to run commands or scripts as root, like SimpleMDM can.

✌🏻