RAM issues of a dell workstation

We have a Dell Precision 7920 Workstation at the institute. It runs Proxmox just fine with its 768GB DDR4 RAM and dual socket.

One day it stopped booting at all after a reboot.

It stayed there with a white system LED with the fans working and the CPU getting hot.

Rebooting did not show any failure signs. The system LED is white, the network LED is green, the PSU light is green.

After talking a while with the technical support, they suggested to order a new motherboard as it seems that the motherboard is defect.

A month later, the motherboard arrived. We replaced it using the Dell instructions here aaaaand…

Investigation

We got the same error. After checking everything, we eventually found a pattern after some trial and error.

Contacting Dell support again was time consuming as well and they guided us to a wrong track by saying that a video should show up on the machine during the first 30 seconds.

This was especially tricky, as the Workstation 7920 motherboard does not have a video onboard, so one has to use the GPU attached to get any video at all.

We found out later, it can take about 180 seconds for the video to turn on with a successful boot and always takes more than 60 seconds.

Due to this, it seems that we wrongly claimed a boot with only a single ram bar to fail. Or maybe we did have the unlikely case to try with the faulty bar. Or maybe we did insert the single bar to the wrong slot. Many things can go wrong with this inspection as I will tell here.

RAM investigation

By removing all RAM bars we found that the blink code of the LED did show the correct one - two orange blinks followed by three white - as shown in this page.

With the new mainboard we tried a few more things regarding RAM as the Dell support did claim that it probably is a RAM issue then. Yet the motherboard does not blink for “Memory failures” but just shows a working white LED without booting. Even when running for hours.

Having only two RAM bars inserted, nothing worked at first. But having the right RAM bars used, the device booted after 3 minutes and did show some funny “*** Dell Internal Use Only ***” Service menu allowing to set a service tag. Of course, that’s fine, as we switched the motherboard to the new one, we do not have a service tag configured yet. Entering the old service tag (double check!) and continuing the boot fixes this.

Yet, one has to watch out to add the bars in the correct order. On the right side (CPU0) of the workstation, the lowest bars have to be filled first.

On the left side (CPU1) the outline is 180° around. The two upper white bars have to be filled first.

It might be, that this was due to some weird behavior, but I did not have a working workstation when using different bars (aka same behavior, white LED, no video output, no boot)

Now it was only a matter of bisecting the RAM bars:

• adding half of remaining bars
  • if it works, add half of remaining
  • if it does not work, remove half of added bars

So it took me the following steps:

1. it did boot with 2 bars
2. did not boot with 6 bars in CPU0
3. it did boot with 5 bars at CPU0
4. did not boot with 5 on right side, and 2 on the left side
5. did not boot with 5 on right side, and 2 others on the left side
6. it did boot with 5 on right side, and 4 on the left side
7. it did boot with 5 on right side, and 6 on the left side
8. checked if adding the last remaining bar works (did not)
9. checked if switching bars helped (did not)

So we found the root cause of this problem. One 64GB RAM bar which did not behave correct.

Fan problem with FB2

So when booting, the Dell workstation does a healthcheck to see if everything works. It failed during this boot with the error that the “FB2 fan failed to respond correctly”.

One can surpass this by entering the boot selection mode prior to the Dell healthcheck to get a working system. When rebooting, we are landing in the diagnostics boot again though.

On a chinese blog, I found someone who also got this error and figured it out:

• the FB2 fan is the HDD fan
• since we replaced the motherboard, the information how many fans are enabled is wrong
• we need to disable HDD fan 3 and 4 in UEFI as our workstation only has 2 HDD fans (seen on the other side of the mainboard)
• after this, everything works well

It seems, that I could have also contacted Dell about this issue, but I somehow distrust them enough that I would search the internet first.

---

English translation of chinese blog

Recently, I started to maintain the servers in my lab (dirty, tired, and miserable +-+), and I encountered the server BIOS I will step in the pits later, so I will™ remember it again.

It took two days to solve the problem of false alarms triggered during the boot autofailure. As an introduction to maintenance, First of all, it is important to understand that when Dell servers of this model are turned on, the BIOS will perform a check of each piece of hardware on the motherboard, i.e., perform a power-on self-check, which highlights the fact that it sends and receives the feedback signal to the pin outlets of each piece of equipment on the motherboard, and when it does not receive the detection signal correctly, it will report an alarm and make a beeping sound. This time, I encountered an alarm problem that the FB2 FAN pin connector did not respond correctly, OK, this is a pitfall.

Solution Idea: First of all, looking at the name, I judged that it should be a fan problem, but strangely enough, what fuck is FB2? What the hell is the fan in here? I had to go to Dell’s user manual, but after reading through the fan section of the manual , it only includes 7 fans, i.e., 4 cooling fans on the front panel (sys0, sys1, sys2, sys3), plus 2 cooling fans on the rear panel (rear0, rear1), and 1 CPU fan (CPU_FAN), for a total of 7 fans.

However, when I performed the actual power-on test, i.e., powering on the chassis to check whether the fans were spinning or not, I found that there was no problem with these 7 fans (in fact, the power-on auto-check also showed that there was no problem with these fans). So the question is, where is the FB2? I had to contact Dell for advice, and finally found that there were five more hard disk fans on the back panel of the chassis.

The source of this is https://www.laitimes.com/article/4e3lq_4uqou.html and https://blog.csdn.net/ZhouZhouMonster/article/details/95526890 though I am not linking to it, as the website looks a lot like malware.

The machine does work fine now and boots correctly. It would not have been needed to replace the motherboard, which caused the funny FAN issues (which are also not explained or documented by Dell itself).