Listing some of the common errors when running the UST and tips to resolve them
FileNotFoundError: [Errno 2]
Sample console error output:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\USER\\.pex\\install\\pycryptodome [...]'
May appear on Windows because of paths > 256 chars.
Tip: create an environment variable named PEX_ROOT with value C:\pex (if the script is run from the C: drive, or change the letter to match). Sometimes a system restart is required for it to take effect.
can't open file 'user-sync.pex': [Errno 2]
Sample console error output:
python: can't open file 'user-sync.pex': [Errno 2] No such file or directory
Tip: make sure you run the python command line from inside the folder where the user-sync.pex is located.
Sample console error output:
2018-01-01 11:49:42 28102 WARNING umapi - UMAPI timeout...service unavailable (code 429 on try 1)
2018-01-01 11:49:42 28102 WARNING umapi - Next retry in 42 seconds...
Tip: These Warning messages are normal. When busy, the server replies with 429 HTTP code, so a retry might resolve it. UST comes with an exponential back-off mechanism to retry the call, increasing the time in between retries. The script stops after three failed retries.
error.user.belongs_to_another_org
Sample log entry (debug mode):
2018-01-01 11:49:42 28102 ERROR umapi.action - Error in requestID: action_1 (User: {'user': 'myuser@domain2.com', 'requestID':'action_1'}, Command: {'createFederatedID': {'email': 'myuser@domain2.com', 'country': 'US', 'option': 'ignoreIfAlreadyExists', 'firstname': 'fname', 'lastname': 'lname'}}): code: "error.user.belongs_to_another_org" message: "Illegal to invite user from another organization's owned auth src"
Tip: the domain used to create the account might not be a claimed/trusted inside your Organizations. A green flag or dot should appear for the active domains in Admin Console -> Settings. If that is not the case, completing the domain claim process can resolve it.
Sample log entry (debug mode):
2018-01-01 12:34:23 13383 ERROR umapi.action - Error in requestID: action_6 (User: {'user': ‘user@domain.com’, 'requestID': 'action_6'}, Command: {'createEnterpriseID': {'email': 'user@domain.com', 'option': 'updateIfAlreadyExists', 'firstname': 'test', 'lastname': 'user', 'country': ‘US’}}): code: "error.user.type_mismatch" message: "The user type requested for the invite does not match the claimed domain type"
Tip: an attempt is made to create a federatedID type of account but the Directory is created for Enterprise use or conversely. Search the attribute user_identity_type inside user-sync-config.yml file. Modify the value according to the type of Directory shown in Admin Console (Settings > Identity > Domains > Directory type value for the given domain).
Sample log entry (debug mode):
Sample console error output:
Failed to execute PEX file, missing compatible dependencies for:
pyyaml
cryptography
cffi
umapi-client
pycryptodome
pyldap
psutil
user-sync
Tips:
a) Check if the Python version you've installed on your system is the 32bit version. Uninstall the 32bit and install the 64bit version, to fix the issue.
b) Check whether the user-sync.pex version you've downloaded from GitHub is a match to your Python version and OS type. For example, user-sync-v2.3-win64-py365.zip is to be downloaded for Windows 64-bit and Python 3.
Using the latest version of Python is not always a good idea. Trying to match the Python version, the .pex was built in the first place, would be great. Follow the suffix of the downloaded .zip file to identify what Python version would work. For the example above (user-sync-v2.3-win64-py365.zip), Python 3.6.5 can be identified.
Error decrypting private key
Sample log entry (debug mode):
2018-01-01 09:52:23 7920 DEBUG umapi - umapi: reading private key data from file C:\path\to\private.key
2018-01-01 09:52:23 7920 CRITICAL main - umapi configuration.enterprise: Error decrypting private key, either the password is wrong or: RSA key format is not supported
Tips:
a) Sometimes, the private key may contain some undated characters or empty lines inserted by mistake. Try removing the empty lines or regenerate the pair private and public key.
b) Do not use umapi_private_key_data attribute if you are running the script in Windows. Instead, encrypt the key and store that password inside Credential Manager.
c) Try an RSA256 /2048bits length private key, if you use a different format.
d) It's possible that you set up the secure_priv_key_pass_key: umapi_private_key_passphrase inside the connector-umapi.yml file. In this case, ensure that the entry in Credential Store for this and associated values match (see below).
No value in secure storage for user...
Sample log entry (debug mode):
2018-01-01 09:52:23 7920 CRITICAL main - umapi CRITICAL main - umapi configuration.enterprise: No value in secure storage for user "someUUIDvalue@AdobeOrg", key "umapi_api_key"
Tips:
a) the Credentials Store entry for umapi_api_key may be missing. Create the entry in Credentials Store. Helping documentation can be found here.
b) It may be that the value has been added to Credentials Store under a different user account. However for the currently connected user this entry is missing. Add it, if necessary, or switch user accounts.
error.internal.exceptionflys / error.unauthorized
Sample console error output:
umapi_client.error.RequestError: Request Error (401): {"lastPage":false,"result":"error.internal.exceptionflys","message":"Failed to exchange token"}
OR
"umapi_client.error.RequestError: Request Error (401): {"lastPage":false,"result":"error.unauthorized","message":"Failed to authenticate provided token"}"
Tip: It may be that in Admin Console -> Settings -> Authentication Settings there is an option chosen other than Easiest for Users (Password never expires). If More Secure or Most Secure option is enabled, it can expire the password of the Technical Account linked to the integration.
To fix the issue, a new integration should be created. Ensure that the metadata is also renewed in the connector-umapi.yml file.
A fix was deployed for it, but it can affect integrations created before October 2018.
Failed to establish a new connection [Errno 10061]
Sample console error output:
ConnectionError: HTTPSConnectionPool(host='usermanagement.adobe.io', port=443): Max retries exceeded with url: /v2/usermanagement/users/someUUID@AdobeOrg/0?directOnly=True (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x00000000027B9630>: Failed to establish a new connection: [Errno 10061] No connection could be made because the target machine actively refused it',))
Tip:
It is related to UST not being able to connect to the public API endpoints. Local settings can prevent access due to firewall rules, proxy blocking traffic, account settings for Internet access, and more.
In some situations it may help adding these two environment variables:
http_proxy and associated value of format http://<host proxy>:<port>
https_proxy and associated value of format https://<host proxy>:<port>
In other cases, allowing access to these endpoints might help:
ims-na1.adobelogin.com:443
usermanagement.adobe.io:443
It can only be resolved locally by clearing access to the above endpoints for the running account.
Sample log entry (debug mode):
2017-07-07 09:01:37 4916 CRITICAL main - Connection to org [...] at endpoint https://usermanagement.adobe.io/v2/usermanagement failed: Unable to authorise against https://ims-na1.adobelogin.com/ims/exchange/jwt: Response Code: 400, Response Text: {"error_description":"The metascopes in the JWT are not a subset of the metascopes in the binding.","error":"invalid_scope"}
Tip: Access the integration you've created at https://console.adobe.io/projects and check the left menu where the APIs are listed. Make sure that User Management API is added as a service (appears in the list).
No valid bindings were found for organization and technical account combination
Sample log entry (debug mode):
2017-07-07 09:01:37 4916 CRITICAL main - UMAPI connection to org id 'someUUIDvalue@AdobeOrg' failed: Unable to authorize against https://ims-na1.adobelogin.com/ims/exchange/jwt:
Response Code: 400, Response Text: {"error_description":"No valid bindings were found for organization and technical account combination","error":"invalid_token"}
Tips:
a) It can be caused because the tech_acct value inside connector-umapi.yml file corresponds to other value than technical account ID inside the integration at https://console.adobe.io. Verify the technical account ID value from the current integration and copy it to this file.
b) It can also be caused because the public certificate from the integration is expired. Renew the private and public key. Then, upload the public key, and replace the old private key with the new key. Verify the path inside connector-umapi.yml file to point to the right file.
c) Check if the Integration is made for the correct Organization. Select the Organization inside the drop-down list located in the upper-left corner at https://console.adobe.io/integrations. Then, verify the technical account ID value for the active integration along with the other metadata (Org ID, secret, and client ID).
SSL: CERTIFICATE_VERIFY_FAILED
Sample log entry (debug mode):
2017-07-07 09:01:37 4916 CRITICAL main - UMAPI connection to org id 'someUUIDvalue@AdobeOrg' failed: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)
Tip: It is because there may be some SSL inspection enabled on the proxy server (local environment setting)
Solution1: Obtain the firewall's CA certificate in PEM format (assume that its name is thecert.crt). If DER format is used, transform it to PEM using openssl command:
openssl x509 -inform DER -in thecert.crt -out thecert.pem -outform PEM
Note: if unsure that the .crt file is already PEM format or not, run these command lines first and see which one fails.
openssl x509 -text -inform DER -in thecert.crt
openssl x509 -text -inform PEM -in thecert.crt
If DER one fails, you already have the file in PEM format so rename the thecert.crt to thecert.pem, otherwise transform it to PEM using the first openssl command line above.
Next, create an environment variable named REQUESTS_CA_BUNDLE and set its value as the path to thecert.pem file.
Solution2: On Windows, in cases where the tool is run from a driver that is different from the one where the OS and Python is installed. In this case, it cannot reach the bundle of trusted root CA certs. Moving the entire script on the drive where the OS exists can be a solution. If it is not an option, the cacert file containing all the trusted root CAs should be used as target for the REQUESTS_CA_BUNDLE env variable. If a proxy inspects the SSL traffic, the content of the root CA certificate must be copied inside the cacert.pem file to verify the certificates.
Note: A default Python install would have the bundle of certificates at C:\Python36\Lib\site-packages\certifi\cacert.pem.
Solution3: disable the SSL inspection on the proxy side for the API endpoints ims-na1.adobelogin.com and usermanagement.adobe.io
Sample log entry (debug mode):
2018-01-01 09:01:37 4916 WARNING ldap - No group found for: Name_Of_The_Group
Tips:
a) This group does not exist in LDAP with that exact name; fix it by adding the correct LDAP name of the group
b) The group is not discoverable under the declared base_dn (see connector-ldap.yml file). Change the base_dn value to contain the mentioned group (it mainly occurs when base_dn points to some OU).
Sample log entry (debug mode):
2018-01-01 11:25:45 1 ERROR umapi.action - Error in requestID: action_1 (User: {'user': 'myuser@domain.com', 'useAdobeID': True, 'requestID': 'action_1'}, Command: {'add': {'product': ['group_name']}}): code: "error.group.not_found" message: "Group my_group_name was not found"
Tip: The user-group group_name in the output above does not exist on Adobe side, so go ahead and create it. If the intent was to set the name of a PLC instead one of a user-group, then check this documentation page for a more visual explanation.
Sample log entry (debug mode):
2018-09-05 10:58:08 96329 CRITICAL main - Connection to org some_Org_UUID@AdobeOrg at endpoint https://usermanagement.adobe.io/v2/usermanagement failed: dlopen(/Users/user/.pex/install/cryptography-2.3-cp37-cp37m-macosx_10_12_x86_64.whl.f77d5cc74b0deef9f1df7eacfe5f5ea57ed94a63/cryptography-2.3-cp37-cp37m-macosx_10_12_x86_64.whl/cryptography/hazmat/bindings/_openssl.abi3.so, 2): Library not loaded: /usr/local/opt/openssl/lib/libssl.1.0.0.dylib
Referenced from: /Users/user/.pex/install/cryptography-2.3-cp37-cp37m-macosx_10_12_x86_64.whl.f77d5cc74b0deef9f1df7eacfe5f5ea57ed94a63/cryptography-2.3-cp37-cp37m-macosx_10_12_x86_64.whl/cryptography/hazmat/bindings/_openssl.abi3.so
Reason: image not found
2018-09-05 10:58:08 96329 INFO main - ========== End Run (User Sync version: 2.3) (Total time: 0:00:00)
Tip: this error was recorded on a macOS High Sierra, when using UST v2.3, using Python 3.7.0. Running brew install openssl inside the Terminal fixed it for this particular scenario.
Could not create person for type 2 or 3 [...]
Sample log entry (debug mode):
2019-07-28 07:17:51 2220 ERROR umapi.action - Error in requestID: action_1 (User: {'user': 'user@claimed-domain.com', 'requestID': 'action_1'}, Command: {'createFederatedID': {'email': 'user@claimed-domain.com', 'option': 'updateIfAlreadyExists', 'firstname': 'First', 'lastname': 'Last', 'country': 'US'}}): code: "error.internal.create_failed" message: "Could not create person for type 2 or 3. Renga result is NOT_ALLOWED, Resource is externally managed and token is missing the GROUP_SOURCE_UPDATE purpose."
Sometimes, the @claimed-domain.com is owned by a different Organization that has set up an Azure or Google connector to manage the synchronization of accounts to Admin Console. This same domain is then entrusted to a different Organization that uses the User Sync Tool to synchronize accounts of format @claimed-domain.com
Cause: The logged message appears when the Sync tool extracts the
user@claimed-domain.com account from an LDAP server to create it in the secondary organization. However, the account is not yet created/sycned into the main organization via Azure or Google connector.
Tip: try to create/sync the user@claimed-domain.com account in the Organization using thee Azure or Google connector, then retry syncing with the UST in the trustee Org.
Unexpected LDAP failure reading [...]
Sample log entry (debug mode):
2018-09-05 10:58:08 96329 6348 CRITICAL main - Unexpected LDAP failure reading group info: {'desc': 'Referral', 'info': 'Referral:\nldap://domain.local/DC=sub,DC=domain,DC=local'}
Possible cause: The group(s) of interest are located in a subdomain, but the host value is one of the root domains.
Tip: Change the host value to one of the subdomains where user groups can be found. If users or groups of interest are located in both root domain and its sub-domain(s), use the global catalog port on the root domain. Modify the groups in the sub-domain(s) to be Universal instead of Global. Example of host value using global catalog: ldap://domain.local:3268 or ldaps://domain.local:3269
In this last scenario, if global catalog port is used, make sure base_dn value takes no value:
base_dn: ""
