Migrating a Database fails due to Storage Issues

Overview

You attempt to migrate a previously symlinked database but encounter issues due to the lack of storage. Your end-users report that they cannot make or delete directories. You receive the following errors in your venture log when attempting the migration to the new volume:

• "Cannot create database <new_db_name>, /NEW_DATABASE_VOL"
• "Minimum free space threshold reached, syncvoltodb exiting."
• "sync_dovoltodb: Failed to sync err=-2"
• "db_mysqlxquery: Query Error! [Error dropping database (can't rmdir './tempdb/', errno: 17)], err 1010, depth 0, cid 23086, pid 18375, time 0
  BAD[DROP DATABASE IF EXISTS tempdb]"

Solution

Due to the space concerns on the current volume, you must fully migrate the database to a new volume. When migrating the database to a new location, WebNative normally automatically creates a new symbolic link for the new location.

The process below will guide you to migrate your data and readjust the necessary symbolic links manually.

1. Verify whether the current webnative database is using a Symbolic Link:
   1. Navigate to your Database Directory.
      1. Default Path: /usr/etc/venture/data
   2. Execute # ls -la and verify that you see the "->" link syntax as shown below:
      
      
      
      
   3. If your database is not currently symbolically linked to an external volume, the procedure below is unnecessary. You can instead move your most recent backup to the new volume, and reference Restoring Database Backups.
2. Login to WebNative Admin and stop the DB Process and DB Daemon
   1. Navigate to Database > Admin > Settings
   2. Select Stop Database and Stop Daemon
      
      
      
      
   3. Note: If your database backs up to a dedicated physical computer and you are moving the content to a new server, consider backing up the /usr/etc and /usr/adm directories now using 3rd-party backup software (eg. Acronis) to capture any existing configuration details.
3. Create the new directory where your database will reside
   1. Verify its ownership and permissions match the current folder.
   2. You can execute # ls -la within the parent directory of the old path to quickly verify the old permissions and folder owner.
      
      
      
      
4. Copy the existing data to the new location
   1. There are many ways to accomplish this move, and the appropriate method for your organization will depend on various factors, such as uptime requirements or the total size of the data to be moved.
   2. Two common Unix-based options to consider:
      1. The built-in Copy(cp) Command:
         1. # cp -rp <current path to DB>/* <new path to DB>/.
         2. Note: If the total size of your database is quite large, it can take some for the cp command to finalize the move.
      2. Rsync and CRON job:
         1. Note: Support for the utilization or configuration of 3rd-party components is out of Xinet Support's scope of support. It is suggested that you discuss the viability of using these tools with your System Administrator.
         2. These 3rd-party tools provide a method for avoiding DB downtime by scheduling the automatic synchronization of content during a safe time window.
         3. Find below an example using Rsync and CRON to perform the sync at 2 AM.
            • Install rsync: # yum install rsync
            • Create a CRON entry to run the command every day during a safe time window.
              • # crontab -e
              • Insert this line: 0 2 * * * rsync -vaf /OLD_DATABASE_DIRECTORY/ /NEW_DATABASE_VOL --delete > /OutputLog.log
                • Note: Replace /OutputLog.log with the Path to write the output of the command to make sure everything is working as expected.
              • Note: You can make sure the crontab is installed by executing # crontab -l
            • This command will keep a mirror image of your current /OLD_DATABASE_DIRECTORY in the target /NEW_DATABASE_VOL
            • On the day of the final migration, run the command manually after stopping the DB process and daemon:
              • # rsync -vaf /OLD_DATABASE_DIRECTORY/ /NEW_DATABASE_VOL --delete
5. Replace the old Symbolic Links
   1. If you had existing symlinks:
      1. Remove any symlinks for the initially failed migration:
         1. # cd /path/to/failed_symlink
         2. # rm -f initial_tempdb_symlink
      2. Remove the existing webnative symlink:
         1. # cd /OLD_DATABASE_DIRECTORY
         2. # rm -f webnative
   2. Create a new Symlink for the webnative database:
      1. # ln -s /NEW_DATABASE_VOL webnative
   3. Change the group to apache:  
      1. # chgrp -h apache webnative
6. Restart the Database and Daemon via the WebNative Admin Portal.
   1. Navigate to Database > Admin > Settings
   2. Select Start Database and Start Daemon.

Testing

After the migration completes, it is suggested that you run a Table Check for signs of any trouble.

1. Navigate to Database > Admin > Table Check
2. Select Check Now
   
   
   
3. If there are signs of failure:
   
   1. Review the check log directly by selecting "View Log" on the right.
   2. If the failures were temporary, and you see that subsequent table checks show that no tables require repair, the migration was successful.
      
      
      
      
   3. If the failures are ongoing, verify that the DB configuration is still valid:
      1. Navigate to Database > Admin > Settings
      2. If the details appear valid, please open a ticket with Xinet Support and provide the following:
         1. Venture Log:
            1. /var/adm/appletalk/venture.log
         2. Xinet System Log:
            1. /var/adm/appletalk/at_log
         3. Table Check Log:
            1. /var/adm/appletalk/check.log
         4. Screenshots of the DB Settings and Table Check UI:
            1. Database > Admin > Table Check
            2. Database > Admin > Settings