How to manage the local administrators group on Azure AD joined devices

See some ongoing confusion when the customers are trying to follow the recommendations in the following official documentation – How to manage the local administrators group on Azure AD joined devices.

The administrators are expecting to see the user account they have added in the Azure AD device administrator role by enabling “Additional local Administrators on Azure AD joined devices” option in Azure AD portal and assigning users to it

 

to show up as a separate item in the local Administrators Properties on Azure AD joined device.

But all they see is something like this screenshot.

Per mentioned documentation:

When you connect a Windows device with Azure AD using an Azure AD join, Azure AD adds the following security principles to the local administrators group on the device:

• The Azure AD global administrator role
• The Azure AD device administrator role
• The user performing the Azure AD join

 

So there will be no separate entry for the user who was added to Azure AD device administrators role since he is part of the role. As you can see on the screenshot of the local Administrators Properties, there is an entry for the user who performed Azure AD join and two more SIDs – one is for Azure AD Global Admin role, other for Azure AD device administrator role (keep in mind the SIDs will be different for each tenant).

Other important thing to highlight –

Updating the device administrator role doesn’t necessarily have an immediate impact on the affected users. On devices where a user is already signed into, the privilege update takes place when both the below actions happen:

4 hours have passed for Azure AD to issue a new Primary Refresh Token with the appropriate privileges.

User signs out and signs back in, not lock/unlock, to refresh their profile

Azure AD Conditional Access policies troubleshooting – Device State: Unregistered

With Azure AD Conditional Access (CA) policies you can control that only managed devices can access resources protected by Azure AD – https://docs.microsoft.com/en-us/azure/active-directory/conditional-access/require-managed-devices#managed-devices

As mentioned in the article above, you might require the devices the sign in is taking place from to be hybrid Azure AD joined.

As explained in this blog – https://jairocadena.com/2016/11/08/how-sso-works-in-windows-10-devices/ the Azure AD Primary Refresh Token (Azure AD PRT) is used during Azure AD CA policies evaluation to get the information about Windows 10 device registration state.

The mentioned blog explains that the Azure AD PRT is initially obtained during user sign into the station. In simple words, if the Cloud AP plugin is able to authenticate on behalf of the user (UPN and password or Windows Hello for Business PIN) to get the Azure AD access token and device is able to authenticate to Azure AD using the device registration state (MS-Organization-Access certificate) the Azure AD PRT will be issued to the user.

If any of these two parts (user or device) didn’t pass the authentication step, no Azure AD PRT will be issued.

So when you see an Azure AD Conditional Access error stating that the device is NOT registered, it doesn’t necessary mean that the hybrid Azure AD join is not working in your environment, but might mean that the valid Azure AD PRT was not presented to Azure AD.

To check if the Azure AD PRT is present for the signed into Windows 10 device user, you can use the “dsregcmd /status” command. Windows 10 OS version 1809 the Azure AD PRT info is stored in the SSO State section:

+----------------------------------------------------------------------+
| SSO State                                                           |
+----------------------------------------------------------------------+
     AzureAdPrt : YES
     AzureAdPrtUpdateTime : 2019-04-03 17:25:24.000 UTC
     AzureAdPrtExpiryTime : 2019-04-17 21:25:54.000 UTC
     AzureAdPrtAuthority : https://login.microsoftonline.com/tenantID

By the way you can use usual /? Switch to get help for the “dsregcmd” command (Windows 1809 and newer versions).

Keep in mind that the Azure AD PRT is a per user token, so you might see AzureAdPrt:NO if you are running the “dsregcmd /state” as local or not synchronized (on-premises AD user UPN doesn’t match the Azure AD UPN) user.

Per my experience, here are examples of what might be the root of Azure AD PRT being absent for the user (will be updating the list as discover more possible root causes):

1. Device indeed is not hybrid Azure AD joined;
2. Local registration state of the computer doesn’t match the records in Azure AD:
   • Azure AD computer object was deleted by Global Admin via portal or PowerShell;
   • Computer was moved out of Azure AD Connect sync scope and was removed from Azure AD by Azure AD Connect;
   • Some services modified the Azure AD computer object and deleted the AlternativeSecurityIds attribute from Azure AD Computer object);
3. CloudAP plugging is not able to authenticate on behalf of the user to get Azure AD access token:
   • If the user is federated, the on premises STS is not reachable or STS do not have WS-Trust endpoint enabled (yes, WS-Trust is still required for Azure AD PRT flow and optional for Windows 1803 and newer registration flow) (for AD FS the WS-Trust endpoint is – adfs/services/trust/13/usernamemixed)
   • The user has recently changed the UPN and is using Windows 1709 or older OS version and can’t get new or refresh expired Azure AD PRT – this issue was resolved in 1803 and newer);

Here are the recommended troubleshooting steps for mentioned above scenarios:

1. To troubleshoot why the computer can’t perform hybrid Azure AD join refer to the following post – https://s4erka.wordpress.com/2018/03/06/azure-ad-device-registration-error-codes/;
2. To better understand if there is a discrepancy between local registration state and Azure AD records, collect and review following info:
   1. “Dsregcmd /status” output on the effected computer, make the notes of the following fields: AzureAdJoined, DeviceCertificateValidity, AzureAdPrt, AzureAdPrtUpdateTime, AzureAdPrtExpiryTime;
   2. Check the Azure AD Portal – Devices blade, see if the station is present in Azure AD and has a timestamp listed in the Registered column, compare with the time in the DeviceCertificateValidity from the previous step. If there is no time stamp in the Registered column, that means that the AlternativeSecurityIds attribute (contains the MS-Organization-Access certificate thumbprint. This is the certificate that was saved to the station during registration process) was removed and the station needs to be re-joined to Azure AD;
   3. You can check if the station has the AlternativeSecurityIds attribute by using the Get-MsolDevice Azure AD PowerShell cmdlet;
   4. Check if the computer object is in the sync scope of Azure AD Connect;
3. To get more clues about user portion of the Azure AD PRT receive process, its recommended to review the following Windows 10 logs – Application and Services Logs – Microsoft – Windows – AAD. These logs contain Operational and Analytic logs. Analytic logs are the equivalent of the Debug logs and are disabled by default. Usually you should be able to get info just by looking at the AAD Operational logs. In the AAD Operation logs look for the events generated by AadCloudAPPlugin Operation. By readying these logs you should get an idea either the STS is not reachable because of the network or protocol issues or Cloud AP is not able to authenticate on behalf of the user due to incorrect credentials or access policies configured on STS that block the authentication attempt for this user;

You can also use the “Get-WinEvent” PowerShell cmdlet to quickly pull latest AAD logs related to Azure AD Cloud AP plugin:

 
Get-WinEvent -LogName "Microsoft-Windows-AAD/Operational" -MaxEvents 20 |
where {$_.TaskDisplayName -like "*AadCloudAPPlugin*"} |
ft TimeCreated,id,KeyWordsDisplayNames,Message -wrap -autosize

Keep in mind that Windows down-level devices do not have Azure AD PRT and they “proof” to Azure AD CA that they are registered by establishing TLS authentication channel using the MS-Organization-Access certificate saved in the User certificate store during device registration. So if the successfully registered down-level Windows device is treated by Azure AD CA policy as not registered, most likely something (firewall/proxy) is messing up with that attempt of the device authentication.

And final thought. In case you need to re-join the Windows current device, make sure to follow the steps in this order to make sure the station really disjoined and will try the clean join process. Also keep in mind that since the computer object is recreated, the Bitlocker recovery keys that you might be saving in Azure AD for this station will be deleted and you will need to re-save them .

1. Open elevated CMD (as local Admin) and issue “dsregcmd /leave”. Elevated CMD is important part, since during the leave flow, the registration service is trying to contact Azure AD and delete the computer object and also it tries to delete the MS-Organization-Access certificate from Computer certificate store, that definitely requires elevated privileges;
2. Open new CMD window and confirm that the local registration state is cleaned and the station is not Azure AD joined by issuing “dsregcmd /status”;
3. Using Azure AD devices portal confirm the computer object is gone, if not, delete it manually;
4. In case you are in Managed environment, you need to run delta Azure AD Connect sync to pre-sync the AD computer object to Azure AD;
5. Restart the station.

RSA SecurID Access SAML Configuration for Microsoft Office 365 issue – “AADSTS50008: Unable to verify token signature. The signing key identifier does not match any valid registered keys”

There recently have been couple cases when the customers who has configured the Azure AD federation with RSA SecureID by following these instructions https://community.rsa.com/docs/DOC-1019 were randomly experiencing the error during users sign in: “AADSTS50008: Unable to verify token signature. The signing key identifier does not match any valid registered keys”.

The issue temporary goes away when the domain is converted to managed and back to federated.
Looking at the mentioned RSA instructions, noticed that the RSA recommends to set up Issuer Entity ID for the SAML response as “urn:uri:o365”.
Since multiple customers most likely are selecting the same Issuer ID (most likely by copying the example), the confusion takes place on Azure AD side what tenant the SAML response is intended to and the token signing certificate validation fails.
The recommended fix is to change the Issued Entity ID on RSA and AAD side to “urn:uri:your_domain” (ex: urn:uri:contoso) to make it unique.

Internal application published via Azure AD Application Proxy access issues troubleshooting

Recently was troubleshooting the issue when the internal application portal page was not loaded (part of the portal was not loaded at all) when accessed via Azure AD Application Proxy (AAD AP). The application in question was Dell Storage Manager web console, but the troubleshooting steps described below are applicable to any application.

First thing checked the Azure AD application settings related to AAD AP – Azure AD pre authentication was used, no custom domain, headers and application body translation enabled, so setup looked pretty standard.

As next step captured the Fiddler trace when accessing the internal application directly and via AAD AP.

In the trace for the AAD AP access see one of the pages fail to load and this error message:

Azure AD Application Proxy
Root cause: The connector did not respond within the timeout period.
Status code:  GatewayTimeout
Url:  https://xxx/messages
TransactionID:  XXX
ConnectorGroupId:  XXX
Timestamp:  9/4/2018 6:50:00 PM

At the same time, the “messages” page is successfully loaded when the application is accessed directly from the corporate network.

Looking closer at the request and response in both Fiddler traces see next.

Request (redacted):

GET https://IntenalHostName/messages HTTP/1.1
Origin: https://IntenalHostName
Sec-WebSocket-Key: =
Connection: Upgrade
Upgrade: Websocket
Sec-WebSocket-Version: 13
User-Agent: Mozilla/4.0
Host: IntenalHostName

Response (redacted):

HTTP/1.1 101 Switching Protocols
Expires: 0
Cache-Control: no-cache, no-store, must-revalidate
X-Powered-By: Undertow/1
Server: WildFly/8
Pragma: no-cache
Origin: https://IntenalHostName
Upgrade: WebSocket
Sec-WebSocket-Accept: pDsDNKGWwSG8=
Date: Tue, 04 Sep 2018 GMT
Connection: Upgrade
Sec-WebSocket-Location: wss://IntenalHostName/messages
Content-Length: 0

In the bad Fiddler see following:

Request (redacted):

GET https://ExternalHostName.msappproxy.net/messages HTTP/1.1
Origin: https://ExternalHostName.msappproxy.net
Sec-WebSocket-Key: nl/CD3hakpNw==
Connection: Upgrade
Upgrade: Websocket
Sec-WebSocket-Version: 13
User-Agent: Mozilla/5.0
Host: ExternalHostName.msappproxy.net
DNT: 1
Cache-Control: no-cache
Cookie: dsmUsername=; JSESSIONID=ZEfQJAHRszfZGXql33h06aRw.vdellem01; AzureAppProxyUserSessionCookie

Response:

HTTP/1.1 504 Gateway Timeout

So the issue seems to be happening when there is a request to upgrade to Websocket.

The Websocket support by Azure AD App Proxy is currently in preview and it was recommended to collect additional logs to see if it can be fixed in the current case.

To enable the verbose Connector logs, it was recommended to make these changes:

1. Locate the installation directory of the connector (should be C:\Program Files\Microsoft AAD App Proxy Connector)
2. Open the file ApplicationProxyConnectorService.exe.config in notepad for edit
3. Add the following section right after appSettings:

<system.diagnostics>
  <trace autoflush=”true” indentsize=”4″>
    <listeners>
      <add name=”consoleListener” type=”System.Diagnostics.ConsoleTraceListener” />
      <add name=”textWriterListener” type=”System.Diagnostics.TextWriterTraceListener” initializeData=”<PATH_WITH_WRITE_PERMISSIONS> \ConnectorTrace.log” />
      <remove name=”Default” />
    </listeners>
  </trace>
</system.diagnostics>

• Make sure to change to a path with write permissions

4. Restart the connector service and reproduce the issue from your PC while capturing the Fiddler trace.

Looking at the logs, found this exception entry:

System.Net.WebSockets.WebSocketException (0x80004005): Unable to connect to the remote server —> System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. —> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

As the next step, tried to access the application directly from the Connector server using the Internet Explorer browser and sure thing the browser complained about SSL error.

Looking closer, noticed that the internal application URL was protected by SSL certificate issued to the host running the application.

As soon as the application URL was changed to server host name on the AAD App Proxy side, the issue was resolved.

Azure AD device registration error codes

Even when you followed the Hybrid Azure AD join instructions to set up your environment (https://docs.microsoft.com/en-us/azure/active-directory/device-management-hybrid-azuread-joined-devices-setup ), you still might experience some issues with the computers not registering with Azure AD.

If you are looking for troubleshooting guide for the issue when Azure AD Conditional Access policy is treating your successfully joined station as Unregistered, see my other recent post – https://s4erka.wordpress.com/2019/04/05/azure-ad-conditional-access-policies-troubleshooting-device-state-unregistered/

To check if the device was joined to Azure AD run “dsregcmd /status” command in command prompt and look at AzureAdJoined value. For the Azure AD registered devices, it should be set to YES.

If the AzureAdJoined says NO, next step will be to collect information from the Application and Services – Microsoft – Windows – User Device Registration – Admin logs.

First thing, try to locate and read the text description in the error to see if it gives any clue.

Below are some examples of the errors and possible solutions to try.

User Device Registration Admin log – EventID 304 or 305 – adalResponseCode: 0xcaa1000e – recommended step is to check the AD FS claim rules per mentioned above article. It is important to have the AD FS claim rules in the described order and if you have multiple verified domains, do not forget remove any existing IssuerID rule that might have been created by Azure AD Connect or other means.

User Device Registration Admin log –wmain: Unable to retrieve access token 0x80004005 – recommended step is to check the AD FS claim rules.

User Device Registration Admin log – EventID 305 – AdalErrorCode: 0xcaa90006 – make sure the computer is able to reach and authenticate to specified in the error text description Identity Provider endpoint.

User Device Registration Admin log – EventID 204 – Error code: 0x801c03f2 (“The device object by the given id (xxx) is not found.”) – make sure the on-premises computer object is synchronized to Azure AD. Run the Delta Azure AD Connect sync.

User Device Registration Admin log – EventID 304 (309, 201 and 233 coming before it) – Error code: 0x801c0021 (Error code: 0x80072efe in EventID 201) (Or in the User Device Registration Debug logs EventID 500 with message “wmain TenantInfi::Discover failed with error code 0x801c0021”) – most likely the network or proxy didn’t allow the connection to Azure AD device registration endpoints or IdP to complete authentication. Also see next error description for the recommended troubleshooting steps.

User Device Registration Debug log – EventID 502 – Error code: 0x80072ee7 (“WinHttpRequest<class DiscoveryHttpRequest>::OnCallback: The callback handling failed with error code: 0x80072ee7”) – most likely the network or proxy didn’t allow the connection to Azure AD device registration endpoints or IdP to complete authentication. Open IE as System Account using “psexec -i -s cmd.exe” and try to navigate to https://enterpriseregistration.windows.net/verifiedDomain/enrollmentserver/contract?api-version=1.2 (replace VerifiedDomain with your domain). You should see the list of device registration service endpoints like this.

If there is a failure, you might want to configure correct proxy settings in the same IE opened as System Account.

User Device Registration Admin log – 0xCAA90022 Could not discover endpoint for Integrate Windows Authentication. Check your ADFS settings. It should support Integrate Widows Authentication for WS-Trust 1.3. (error message is self explanatory). In case your IdP is not AD FS consult your IdP documentation.

User Device Registration Admin log – 0xCAA9002b with this error from ADAL – ADALUseWindowsAuthenticationTenant failed, unable to perform integrated auth. Check your STS settings. It should support Integrate Widows Authentication for WS-Trust 1.3. (error message is self explanatory). In case your IdP is not AD FS consult your IdP documentation.

User Device Registration Admin log – 0x801c001d. Failed to lookup the registration service information from Active Directory. Recommended to check the Service Connection Point settings in on-premises Active Directory.

Sometimes the error description of the User Device Registration Admin log event is not providing enough information and you have to enable the User Device Registration Debug log to get more information.

To enable debug logs open Event Viewer – check “Show Analytic and Debug Logs” and browse to Application and Services – Microsoft – Windows – User Device Registration – right click on Debug log and select Enable log.

To trigger the device join attempt you have to open Command prompt as System account (you can use Sysinternals PsExec – psexec -i -s cmd.exe) and issue “dsregcmd /debug /join” command. After that disable the Debug log, check the Admin logs and if still the error description is not informative go to Debug logs.

Example 1:

Log Name:      Microsoft-Windows-User Device Registration/Admin
Source:        Microsoft-Windows-User Device Registration
Date:          2/9/2018 10:17:49 AM
Event ID:      304
Task Category: None
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      XXX
Description:
Automatic registration failed at join phase.  Exit code: An unexpected internal error has occurred in the Platform Crypto Provider.

User Device Registration Debug log –

Log Name:      Microsoft-Windows-User Device Registration/Debug
Source:        Microsoft-Windows-User Device Registration
Date:          2/9/2018 10:23:30 AM
Event ID:      500
Task Category: None
Level:         Information
Keywords:     
User:          SYSTEM
Computer:      XXX
Description:
wmain: failed with error code 0x80290407.

Most likely this error is an indication that the TPM is not supporting Azure AD join requirements (https://docs.microsoft.com/en-us/windows/security/hardware-protection/tpm/tpm-recommendations ).

Next steps for this particular issue I would recommend for these stations are:

• Ensure the TPM is in 2.0 mode. You will find this setting in the BIOS.
• As a last resort, disable TPM in the BIOS, so Azure AD Join process uses software-based keys.

Example 2:

Log Name:      Microsoft-Windows-User Device Registration/Admin
Source:        Microsoft-Windows-User Device Registration
Date:          4/17/2018 12:44:10 PM
Event ID:      304
Task Category: None
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      XXX
Description:
Automatic registration failed at join phase.  Exit code: Keyset does not exist. Server error: empty.

After running dsregcmd /debug /join see following in the output:

DsrDeviceAutoJoinFederated failed with -2146893802
wmain: failed with error code 0x80090016.

Most likely this error indicates that the machine was imaged from the already Azure AD registered computer. Also it might indicate the TPM issues (see the TMP troubleshooting steps mentioned above).

If the fist is true, try renaming the “C:\ProgramData\Microsoft\Crypto\Keys” folder and re-running the dsregcmd /debug /join.

Example 3:

Log Name:      Microsoft-Windows-User Device Registration/Admin
Source:        Microsoft-Windows-User Device Registration
Date:          5/16/2018 8:44:03 AM
Event ID:      305
Task Category: None
Level:         Error
Keywords:     
User:          SYSTEM
Computer:     XXX
Description:
Automatic registration failed at authentication phase.  Unable to acquire access token.  Exit code: Unspecified error. Server error: AdalMessage: ADALUseWindowsAuthenticationTenant failed,  unable to preform integrated auth
AdalErrorCode: 0xcaa9002c

This error usually indicates an issue with connecting to AD FS farm. Check if Windows Integrated Authentication is enabled for Intranet, is working correctly for Intranet and WSTrust windows endpoints are enabled in AD FS.