This never was supposed to be serious but when you are having fun, serious and joking around can mix and match.
This is the summery write up and details of my USB raid project to date.
What started out as a “fairly” simple storage server with 8TB of storage turned into the all ridiculous USB-RAID hosting 20TB array via Fibre Channel
Why did I start out?
I started out making a SAN storage server back in 2014 because I wanted storage solution for my old slow server. I originally wanted to save files for distribution over my university’s residential local area network.
My first attempt was to try using a PCI-X card and four 1TB WD blues and two second hand disks. I had a limitation of 2TB logical volumes on the card so I made three and used windows software raid to stitch them together.
How this all worked was less than satisfactory as read and writing speeds were 5MB/s . I thought I needed something better. I had picked up some 4Gbt fibre channel cards second hand and wanted to see if I could beat 1Gbt. I quickly managed to source another desktop PC to test my cards which seemed to function after boot but nothing was configurable with out a target config. After learning that OpenFiler supported Fibre Channel target configs I started out playing with a configuration which shared a 200GB SATA disk.
OpenFiler functioned okay more or less it did the job but configuration was patchy and not all settings were always applied.
I was happy enough so I started buying 4TB hard drives. This is where the USB idea popped up.
I wanted **_all_** my SATA ports. The boot OS was so simple it fitted on less than 8GB. The idea was if I can boot a live Ubuntu distro off my flash drive I could boot OpenFiler. I waited about a month for a 64GB USB3 drive to ship from the US and by then I started filling drives and knew I would need it very soon.
All went without a hitch I transferred my OpenFiler OS to the flash drive and it booted no problem. I was unhappy that the drive had to hang out the back of the case and I had one situation where I came very close to breaking the USB drive off the back while I temporary had it setup in my sisters flat.
To mount it inside the case, I went about cutting an old bracket and screwing it inside the case as shown above.
This went without problems running 24/7 for about 3 months until I finally ran out of disk space AND SATA ports.
I went off and ordered a RAID card and a PCIe riser card.
Already regretting my choice of motherboard I installed both cards and after several reboots and reconfiguration I realised there was a problem and a very big one.
Backup backup backup backup! OMG JUST BACK IT UP!
My hours and hours of work were lost in a simple reboot. The flash drives partition table was gone and several hours of trying to recover it were wasted.
No important data was lost from my storage but for about a month i couldn’t access it due to OpenFiler having a poor application of fibre channel.
My flatmate being in IT he was able to mount the file system locally until I could repair the OS to a usable state.
OpenFiler had been abandoned not only by me at this point but by their developers 😂. I made the decision to change to stock Debain and run the same tool SCST which OpenFiler had used just this time I had all the configuration in my hands.
My choice of OS drive was a single crappy 8GB USB drive given to me by uni. this proved to be far too slow in that simple IO response queries which were processed by the OS tended to time out the storage.
at this point i had spent another few months pissing around trying to get the silly thing to work.
I joked to my flatmate “why don’t i just put the stupid thing onto USB RAID” and thus out of pure silliness and having plenty of drives…
The first USB RAID was born.
i had shared the logical disk over to my desktop to run tests and my experience using one USB bus was not ideal
It was clear I was maxing out my bus for the USB2 RootHub performance was far better than a single disk but I knew I could do better.
By then I had migrated the OS to the USB raid and my storage was functioning without issues once again after about a full year of maintenance.
I added an extra hub to a different bus, specifically the one which served the USB3 ports. With this I was able to achieve over two buses Sequential r/w 63.5 MB/s, 27.3 MB/s http://pastebin.com/ni4ksCXt
I was happy at this point so I started working out how to put the thing inside the case.
I was frustrated at how the case for the white USB hub was and I thought screw it I’m modding this. Out came the trusty soldering iron and tools and I started rotating each port
Things were getting bigger and bigger
and now with 11 drives and 22GB raw, I was in states of hysterics when ever I was working on it. I knew at this point I had created something completely crazy. Each time there was an issue I just made something completely over the top to work around it.
Interestingly enough most systems don’t post with more than 10 USB drives attached so after powering it up for the first time to take the iconic photos I realised I had to manually plug in the hubs after the bios posted.
Over engineering craziness to the rescue! I’ll just use some relays to power it on once the bios has finished posting.
How this works is the micro-controller chip on this custom board listens to the motherboard debug header which emits post bios codes. When it has finished the micro then switches the power through the relays powering on the USB-RAID just in time for grub bootloader to chain-load across to it.
My last tweak to the USB RAID was switching to F2FS which allowed for better file management and apparent less wear and tear on flash based file systems.
My system has been running since June 2016 without any errors.
My current project to date is to make a USB-RAID card card which piggybacks onto a Startech 4port PCIe USB3 card with 4 dedicated USB ports (each with their own bus). I have purchased USB hub controller chips online to give me 16 ports total. Only thing left to do is do up the PCB which due to current funds and work hours has been slow.
I will post more updates as my card develops.
love the blinkies (potato shutter only shows the drives with slow flashing activity LEDs)