Troubleshooting Article - Xinet Search

Overview

This article explains how to troubleshoot issues related to searching in Xinet. Some of the common symptoms related to this topic are:

• Search is returning different results than expected.

• You search in a folder with subfolders using the "Search in current folder only" option but see results from other folders.

• You see "Solr Schema update is needed for current search features" in the Solr UI.

• You see the following error in the server logs, "org.apache.solr.common.SolrException: SolrCore 'file' is not available due to init failure: Exception while loading DataImporter."

Workflow

Instructions

When an end-user reports any issues with Xinet Search, knowing which search field is being used (to determine whether the issue is related to Solr), the search parameters used, and the version of Xinet the issue appears on can help speed up the reproduction and troubleshooting process. Before proceeding, collect these details from the end-user.

1. Reproduce in a new test site running the latest release
2. Resolving issues with "Current Folder only" search results
3. Test Disabling Solr Search
4. Reproduce in a New Volume
5. Check the Solr DB Status
6. Test Disabling Facets
7. Manual Solr Query Testing
8. Attempt Solr Schema Resync
9. Handling the "SolrCore 'File' is not available" error in the Venture logs
10. Check Solr Folder permissions and ownership
11. Check the Venture Logs

Reproducing in a new test site running the latest release

It is important to confirm whether or not this is reproducible within a new test site or not. Due to the number of potential configurations and index states, ensuring that the issue is not isolated to a particular file or server is always the best first step.

If this behavior is reproducible within a new test site running the latest release of Xinet, we will want to engage the Development Team to investigate further. Xinet Support can assist with validating your testing steps and submitting this request on your behalf.

<supportagent>

If a customer reports issues with search, reproduce the behavior reported on the latest release of Xinet before proceeding with creating the Jira. For guidance on setting up a new local test site, see Creating a test environment for Xinet. 

See 7.8.4 Reproducing Defects: Staging and QA Environments for specific details on Engineering's reproduction requirements.

If the issue is reproducible within QA: 

1. Review the Venture Logs:
   1. Collect any Search Error Codes.
   2. Identify whether the issue is isolated to any search patterns.
2. Submit a CDDS to the XINET Jira project, providing all relevant details.

Note: Reference the guidance within 7.8.3 Escalating to Engineering and SaaS Ops.

</supportagent>

Resolving issues with "Current Folder only" search results

If the trouble is specific to the "Search in current folder only" search filter, this is related to a known defect addressed in the 2019.1 release of Xinet. 

See "Search in current folder only" returning files outside the current folder.

Test Disabling Solr Search

Before making any changes to the Solr DB, it is suggested that you confirm whether fully disabling Solr resolves the trouble or not.

If the trouble is not resolved by disabling the Solr DB, consider whether the issue you are experiencing is actually isolated to Solr DB. It is suggested that you verify whether the behavior can be reproduced within a new test site running the latest release to help isolate further.

Reproduce in a New Volume

As an initial check for Solr-specific search issues, consider verifying whether the issue is volume-specific.

1. Create a New Volume
2. Assign User Privileges to allow access
3. Upload a known good test file to the volume.

If the trouble is not reproducible within this new volume, the content within the affected volume may be responsible for the trouble. 

If you can reproduce the trouble in the new volume, it indicates that the trouble is likely system-wide. Use the sub-sections below to review the status of Solr DB and isolate this further.

Check the Solr DB Status

1. Use the SolrUtil to check and resolve any issues in the DB automatically:
   1. From the command line, execute the following:

      # /usr/etc/venture/bin/solrutil -checkdb

   2. If it finds any errors, you can resolve many of these errors by executing the following command to reimport the WMV and update the Solr DB:

      # /usr/etc/venture/bin/solrutil -post

   3. See Using the Solr Utility (solrutil) for more detail on using this utility.
2. Check the Java Memory usage of Solr
   1. In the Solr Dashboard, verify the current usage.
      1. Visit http://Xinet-Server-IP:15969/solr/#
         
         
         
         
   2. If the current memory usage is hitting the upper limit, consider increasing this within your solr.conf:
      1. Locate the following and adjust it based on your requirements:
      2. javaparms=-Xms2048m -Xmx8192m
         1. Xms = Minimum Memory
         2. Xmx = Maximum Memory

Test Disabling Facets

Within the Xinet Admin's Solr Search Index panel, you can temporarily disable all available Search Facets to confirm whether the trouble is related to a particular item.

1. Navigate to Databases > Admin > Searching
2. Within the "Built-In Facets Enabled" section:
   1. Deselect all the facets.
      
      
      
      
   2. Click Save Changes
3. Once disabled, attempt to reproduce the issue.
   1. If the issue is not reproducible, you can re-enable each facet one by one to isolate which item is responsible for the trouble.
   Note: If you have isolated the behavior to a particular facet, please create a ticket directly with Xinet Support to investigate further.

Manual Solr Query Testing

Seeing unexpected Solr search results can often be traced back to a mismatched file count between the Solr DB and the Venture DB. You can use the steps below to verify that the two databases match as expected.

1. Verify the Solr DB count

   1. Navigate to http://Xinet-Server-IP:15969/solr/#/file/query 

   2. Click on the Execute Query button to search all files.

   3. Collect the total numFound count.

2. Verify the Venture DB count

   1. Connect to the Venture DB via the MySQL console:

      1. # /usr/etc/venture/bin/mysql -u root -p

   2. Run the following query:

      1. mysql> select count(*) from file where online = 1;

3. Verify the two results have a matching count.

   1. If the count is mismatched, the Solr DB may be out of date. 

   2. Execute the following commands to update the Solr DB and Solr Schema:

      # /usr/etc/venture/bin/solrutil -post
      # /usr/etc/venture/bin/solrutil -schema

Attempt Solr Schema Resync

If, when visiting the Database Searching section of the WebNative Admin portal, you see an alert stating that the "Solr Schema update is needed for current search features," you can use the SolrUtil to attempt to update the schema automatically.

1. Execute the following command:

   1. # /usr/etc/venture/bin/solrutil -schema

   2. Note: This may take some time to finalize.

Handling the "SolrCore 'File' is not available" error in the Venture logs

If while reviewing the Venture Logs you identify the following error, "org.apache.solr.common.SolrException: SolrCore 'file' is not available due to init failure: Exception while loading DataImporter.", verify whether the following files exist in the filesystem:

• /usr/etc/venture/solr/server/xinet/file
• /usr/etc/venture/solr/server/xinet/file/core.properties

If the files do not exist, Rerun your Xinet Portal Installation Script to reinstall Solr.

1. 1. Execute the following as root to run the installation script:

      /path/to/installation/package/installportal

   2. Note: If you no longer have your installation package, you can download a fresh copy of the Xinet Portal installation package from the downloads section of the Xinet Reseller Portal.

Check that Solr Folder permissions and ownership are correctly set

If you have reinstalled Solr and the trouble persists, please verify whether the ownership and permissions assigned to the Solr Folder are valid.

If they are not, please reference the steps within Portal Assets not Displaying in Pilot dashboard due to Solr DB Failures to fully rebuild the Solr folder as new.

Check the Venture Logs

If previous steps fail to resolve the trouble, the Venture Logs can help isolate the trouble further. The venture logs can provide additional insights into the back-end search parameters being used and why you are not getting the expected results.

You can review these from the WebNative Admin: Logging > Database

If the trouble is isolated to a particular Portal Site, you can temporarily enable the Debug logging for the specific portal site to gain additional insights into the reason for the failure on the site:

1. Portal > Edit > Server Details > Server Defaults
2. Select "Enable portalDI debugging."
   
   

Using these insights while reproducing the trouble can help identify potential reasons for failure:

• Most importantly, verify that no Search Error Codes appear within the logs.
  
  • Most often, these will be the result of improperly formed search query strings, as seen below:
  • 
• If you only see the search failures with keyword searches, verify whether filenames show the same result.

If you cannot identify a clear reason for the failure within the Venture logs, please create a ticket with Xinet Support and attach the collected Venture logs.