10th Sept 2013
RAID Firmware Updates
This message was sent to all clients on NLCKVME5-1 and ATLCVZE5-4. We have recently discovered serious bugs in the firmware on the RAID cards powering these nodes. As such, we have had to downgrade the firmware. Unfortunately it requires a reboot for the changes to take effect. Due to the serious nature of the bugs, we will be rebooting these ...
10th Sept 2013
NLCVZE5-2 Filesystem Repair
This message was sent to all clients on NLCVZE5-2. It seems the filesystem issues that we saw on NLCVZE5-1 have cropped up on NLCZVE5-2 as well. We are diligently troubleshooting this issue, leaning toward it being a RAID firmware problem. We have downgraded the firmware and will be running fsck on the filesystem shortly once we rule out as many ...
2nd Sept 2013
NLCVZE5-1 FSCK
NLCVZE5-1 has gone into another kernel panic. We emailed everyone yesterday that this might happen again. We're running fsck again, although it probably won't fix the issue permanently. We should have another CVZ-E5 node in the Netherlands tomorrow, at which point we will move everyone over to it as promised. It seems to take a couple days for the ...
1st Sept 2013
NLCVZE5-1 Filesystem Errors
This message was sent to all clients on NLCVZE5-1. As several of you are aware, this node has been having some filesystem issues over the past few days. At this point, we are not able to fix the issues in a timely manner (i.e., without another reboot and fsck, which doesn't seem to be working anyway). We are trying to get a replacement node online ...
