Michael de Silva's Blog

Software Engineer. Rubyist and Roboticist.

Michael de Silva's Blog

Software Engineer. Rubyist and Roboticist.

TIL Setting up my FreeNAS 11 Xeon Server

20393783 109952109648900 6213395819798724608 n

Earlier in June, I set about designing and setting up a FreeNAS server, in preparation to migrate away from my existing dual-Synology RAID based system(s) in place.

My primary Synology unit is a 5-bay DSM1515+ with 5x WD Red Pro 4TB drives, configured with a RAID5 array. This is actually a mistake that I initially made when setting this up, driven at the time by cost motivations to provide the most 'available space' with 2-drive failure redundancy.

I would also replicate and backup data to a second 8-bay DSM1815+ with 6x WD Red Pro 4TB drives. This unit would auto-power on between 1-6am to run all the backups and syncing/replication.

Why RAID5 should never be used.

What I did not know at the time, and really should have, was that RAID systems are typically prone to failing at the time of resilvering (or rebuilding) an error at the time of recovery. Assume you have a RAID5 array, and a single drive dies; you insert a replacement drive, and it is at this time that during the re-build a second drive may very-well likely die. You're now left with a DEAD RAID5 array.

There's also something called 'URE' errors or Unrecoverable Read Errors. Interestingly enough, manufacturers tend to report these errors as <10 in 10^15 which is the same as saying 1 error in 10^14 — which means you may get a URE read error every 12.5TB of data. Given that my RAID5 array had ~16TB of usable space, this was another serious point of failure in my system.

In hindsight, I should have bitten the bullet, and gone with RAID6 even though it would have left me with only 12TB of usable space and 3-drive failure redundancy. Even though you loose 50% more space to parity data, it is oh so worth it compared to RAID5.

Why I chose ZFS

In my first build log, I detailed my reasoning for choosing ZFS

I was keen on ZFS and that was the 'priority'; as far as operating systems go, FreeBSD has a track-record of being solid, considered to have a more robust network stack; it's Unix. Had there been a linux-alternative, I may have gone with that.

What really resonated with me though, was the popularity of ZFS (FreeNAS) on YouTube amongst the tech community. @tekwendel has also covered ZFS in quite a few videos, including one where he ends up recovering data from a 'dead-array'.

I basically 'primed' myself for the setup process by watching this excellent walkthough by Tom @ Lawrence Systems and also referred some other videos on the topic.

You can find my build log(s) below and a list of various aspects that I ended up touching on with regards to getting my FreeNAS server up and running.

Build log: https://linustechtips.com/main/topic/810276-freenas-server-build/ Build log (clone): https://forum.level1techs.com/t/serious-freenas-server-build-3d-printing-fun

Tips & TIL from Setting up FreeNAS 11

Confirming ECC is Operational

Ensure your memory is listed with a Total Width of 72 bits, this confirms ECC is working correctly. Non-ECC RAM will run at a Width of 64 bits.

[freenas ~]$ dmidecode -t 17
Handle 0x005C, DMI type 17, 40 bytes
Memory Device
    Array Handle: 0x0058
    Error Information Handle: Not Provided
    Total Width: 72 bits
    Data Width: 72 bits
    Size: 16384 MB
    Form Factor: RIMM
    Set: None
    Locator: DIMM_B1
    Bank Locator: NODE 1
    Type: DDR4
    Type Detail: Synchronous
    Speed: 2133 MHz
    Manufacturer: Micron
    Serial Number: 16EXXXXXX
    Asset Tag: DIMM_B1_AssetTag
    Part Number: 36ASF2G72PZ-2G1B1
    Rank: 2
    Configured Clock Speed: 2133 MHz
    Minimum Voltage: Unknown
    Maximum Voltage: Unknown
    Configured Voltage: Unknown

Encryption (of ZFS Volume)

Your CPU will need to support AES-NI; if no crypto hardware is installed, without hardware acceleration, there will be about a 20% performance hit for a single disk.

The Intel Xeon E5-2620 v4 CPU in my FreeNAS Build supports this.

Enabling Virtualisation

[freenas ~]$ sudo cat /var/run/dmesg.boot | grep VT
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr

Ubuntu Install, Fixing GRUB for UEFI (Post-install)

https://forums.freenas.org/index.php?threads/how-to-boot-ubuntu-desktop-vm-in-uefi-mode.53863/

sudo add-apt-repository ppa:yannubuntu/boot-repair
sudo apt-get install -y boot-repair && boot-repair

Configure APC UPS For Power Management

https://www.cyberciti.biz/faq/how-to-install-apc-ups-on-freenas-server/

[mdesilva@freenas ~]$ dmesg | grep -i --color 'american power conversion'
ugen0.5: <American Power Conversion Back-UPS XS 1400U  FW926.T2 .I USB FWT2> at usbus0

[mdesilva@freenas ~]$ upsc ups@localhost
battery.charge: 100
battery.charge.low: 10
battery.charge.warning: 50
battery.date: 2001/09/25
battery.mfr.date: 2016/06/23
battery.runtime: 1988
battery.runtime.low: 120
battery.type: PbAc
battery.voltage: 27.1
battery.voltage.nominal: 24.0
device.mfr: American Power Conversion
device.model: Back-UPS XS 1400U
device.serial: *********324
device.type: ups
driver.name: usbhid-ups
driver.parameter.pollfreq: 30
driver.parameter.pollinterval: 2
driver.parameter.port: /dev/ugen0.5
driver.parameter.synchronous: no
driver.version: 2.7.4
driver.version.data: APC HID 0.96
driver.version.internal: 0.41
input.sensitivity: medium
input.transfer.high: 280
input.transfer.low: 155
input.voltage: 228.0
input.voltage.nominal: 230
ups.beeper.status: enabled
ups.delay.shutdown: 20
ups.firmware: 926.T2 .I
ups.firmware.aux: T2
ups.load: 19
ups.mfr: American Power Conversion
ups.mfr.date: 2016/06/23
ups.model: Back-UPS XS 1400U
ups.productid: 0002
ups.realpower.nominal: 700
ups.serial: 3B1625X14324
ups.status: OL
ups.test.result: No test initiated
ups.timer.reboot: 0
ups.timer.shutdown: -1
ups.vendorid: 051d
comments powered by Disqus