Corrupt ibdata1 - RAID i/o error

SUGGESTED

Simply Accounting 2012

Data stored on RAID 5 volume. 1 disk is failing, but working. Rebuilding array failed.

Last successful backup is a few months old. 

Database can be accessed and used, but ibdata1 cannot be copied due to the i/o error.

Question: Since Simply will run, is there a way to export ALL DATA going back ONLY until my last backup?

Plan of action:

1. Recover old backup copy to another machine

2. Export latest data form working Simply from date of backup to today

3. Import new data into the recovered backup database

4. Win?

Any thoughts on this?

  • 0
    SUGGESTED

    Based on what my IT guy told me, the power of a RAID system is that if one drive fails, you can separate the RAID, then remove the bad drive and replace it, then resync.  The Good drive will then copy the data to the new drive.

    Alternatively, you can split the RAID and copy the files from the good drive to another computer and test the validity of the data.  If it opens ok, you should be alright but that doesn't mean there was not some damage done to the files.  Sometimes I have heard it is necessary to put the good drive into another computer to copy the files.

    Unfortunately, there is no easy way to take some data from one file and add it to a backup as you suggest.  Technically through a data repair process it is possible but it is tedious and very time consuming.  There is no option on the Simply user interface.

    Have you tried using File, Save As...?

    Were the backups done in the past from within the program or using some server back up method?  If the server, try File, Backup if File, Sage As...

    If you can get the data to copy properly, it might still be necessary to do a data repair but it might be possible to rebuild the file if you can copy it.

  • 0 in reply to Richard S. Ridings

    Hi Richard

    Thank you for your reply. Unfortunately the RAID resync failed. That's the problem. And the file is there, but cannot be copied as the RAID parity is lost and there is an I/O error on the drive. So reading the whole file for a copy is not possible. What's interesting is that the OS is running on another volume of the same RAID virtual disk and this is the ONLY file of the entire volume that I cannot copy. Go figure eh?

    If I place the drive in another computer, unfortunately it will no longer be with the other drives in the RAID volume and may not have all pieces of the file. I believe it won't even be recognized as a file system without the other RAID members and the RAID controller to maintain the data consistency. I could be wrong as my only experience is with Linux and RAID MDADM  where this was the case. IN my situation I'm using a Dell server, Windows Server 2008R2 and a PERC S100 RAID controller.

    So the backups were being done with a third party backup tool that just copied the data folders. Not the Sage backup utility. When I tried to use the sage backup utility in Single user mode it claimed a missing file and could not continue. I'm assuming it couldn't read the whole ibdata1 file and that's the file it was "missing". 

    I've not tried a "Save as". I will have a look at that option. 

    This weekend will be a call to Dell and a call to Sage and see who has a better suggestion. I just thought I'd try here to see if anyone has come across anything like this. Worst case maybe Sage can guide me through a database connection and I can use some SQL queries to pull all data since the last backup. That's a long shot but might be an option.

    Thanks again!

  • 0 in reply to jbost

    Based on your version, you will get limited help from Sage if any, so Dell may be a better bet.  They might be able to help you get the drive to a different computer.  Sage may want you to purchase a current version because they only support 2019.  As of Sunday, support for 2018 will be gone.

    If you can't copy the file, you may have some problems upgrading it, but during the upgrade, between versions (twice) the program will do rebuilds of the database and if they are physically moved to different section of the hard drive, you may be able to get it upgraded properly.  Just be wary of the licensing they will be selling you (perpetual vs subscription - former you own it, except payroll, the latter you are renting and if you stop paying, you get put into read-only mode).  You will also have to download the Sage 50 2015 Trial version to do part of the upgrades to that version.

    Based on which type of RAID level you have, according to the manual here you may only have virtual disk support, not true mirroring.  I was assuming mirroring/redundancy so that you can remove one drive that still works and possibly still get the data off it.  Dell will likely be able to help you sort out which setup you have.

    I can use some SQL queries to pull all data

    This is what I would likely try if I was able to get my computer connected to the network.  Then I would try to rebuild and create a new database.  As I have indicated before, there is no easy way to get and merge data since the last backup.  It would be better to go after all of it, if it could be read.

    Good luck, you have a tough one.

  • 0

    I would try some of the command line COPY parameters, maybe  /B?   or XCOPY /C    and / or run one of the linux distros to boot from a USB to attempt to read the RAID outside the OS.  You might be able to read NTFS and copy it onto a FAT32 filesystem on a second USB.

    Odd that Sage 50 will run, but it's built-in backup / copy won't run - MySQL (at least used to) read ibdata1 and require a valid checksum before allowing the client to connect.  

  • 0 in reply to Richard S. Ridings

    Hey thanks for your reply again. Yes I'll likely contact Dell first. Doesn't matter about warranty. At this point I'm willing to pay for any support request or warranty costs. And we do have a copy of the latest version of Simply. That upgrade was being planned for the future. This was a verification of backup before that. 

    Okay I'll contact Dell and report back to share my experience! Still open to suggestions though.

  • 0 in reply to jbost

    I have an idea, but I am not sure if it should be tried before you try to do anything physical with the drives or not.  If you want to give me a call, we can talk about it. (519) 766-9942

  • 0 in reply to Richard S. Ridings

    Hi Richard,
    I might hit you up if you're around today. I'll PM you for details. 

    Thanks

  • 0 in reply to RandyW

    I've been trying Robocopy a lot. I tried your suggestions with xcopy and copy, but no luck. I also tried rebuilding the RAID again and it failed. Thank you for the recommendations though!

    I contacted Dell, but at this point I'm not sure how much they can support. I'll try Sage first. It's possible that the checksum check isn't implemented in Simply 2012 which is what can currently access it.

    As for the Linux mounting and copying I might try that as a last resort. I did try Hiren's Boot CD and doing the same, but no luck. I know it's running a windows PE, but it was at least outside the OS on the drives. 

    One thing I was thinking, when I tried to rebuild the RAID twice I was using the hotspare without removing the damaged drive. I wonder if the PERC software was still trying to rebuild using data from the bad drive instead of focusing ONLY on the 2 good drives. So perhaps, I should be pulling the bad drive and trying to rebuild from the two good drives to the global hot spare. I know that's how you would do it on a hot-swappable server, but I figured, why pull the bad one if I already have a global hotspare in there? 

    That might be a question for Dell once I pay their $1100 for 2 hours of best attempt service :/