Digital Spaceport

Cryptocurrency => Chia => Hardware => Topic started by: Creative on October 11, 2021, 12:45:08 PM

Title: Dual plotting issues, one of the nodes underperforming
Post by: Creative on October 11, 2021, 12:45:08 PM
Hey there,

I´m plotting in a Server with 256GB of ram 2x10c 20t processors, with two ramdisks and two NVMEs. I open two instances of Madmax and assign one processor to each one in the task manager.

The instance with processor 1 does the job in 59 minutes but the processor 0 does it in 69-75 minutes, it is a huge difference considering it is exactly the same hardware for both, even the NVMEs are the same.

I have tested some time in Ubuntu and both instances were done in 47 minutes (no difference) but I had to switch to Windows because I was having final disks incompatibilities issues, so this meanse that in terms of hardware everything should be the same time.

One doubt I have is when creating two ramdisks there is no way to saparate nodes, I mean, one ramdisk created with ram in node0 and the other ramdisk created with ram in node1.

Does anyone have any idea on how to solve my main problem?

Thanks in advance.
Title: Re: Dual plotting issues, one of the nodes underperforming
Post by: Administrator on October 11, 2021, 01:22:38 PM
Windows is hands down, without any doubt, proven slower for madmax. I have tested that extensively.

https://digitalspaceport.com/is-ubuntu-or-windows-faster-for-chia-plotting-with-mad-max-plotter/

Here is a very in depth article on testing around that.

Plot in Ubuntu is your main speed optimization point. I would also recommend a single plot at a time and using the 243 for the tmpfs size. Should get your plots as low as 1600s on E5-2667v2 class processors.

What processors do you run?
Title: Re: Dual plotting issues, one of the nodes underperforming
Post by: Creative on October 11, 2021, 01:39:55 PM
Thanks for replying.

I´ve been testing in Ubuntu and I managed to get 2 plots in 47 minutes dual plotting or one plot in 29 minutes single plotting, but my issue was that I spent almost 10 days trying to fix an issue with my final mechanical SAS disks. Ubuntu for some reason wont accept them, not even let me format them or do anything to it claiming an input and output error.
Because of this I ended up switching to Windows and no loosing any more time. I really would want to get both windows instances times the same, because windows time with one plot is near 50 minutes.

My processor is 2x E5-2650 v3 running in a DL380 G9 server with 256 of RAM. I think MadMax is underperforming a lot for me.

Could you please give me some advice on what could I do to get the same time in both instances?

Regards
Title: Re: Dual plotting issues, one of the nodes underperforming
Post by: Administrator on October 11, 2021, 01:57:30 PM
You are blaming MadMax, but the issue is Windows. There is not any way around that fact, Windows is just much much slower with the version of MadMax that is compiled for it and will give you times that vary for plotting in stagger in a larger fashion then Ubuntu. I would suggest trying to adjust threads up or down a few and test, but that probably is wasting time for the actual savings. I have not gotten any provable faster times on the hardware I run with staggering MadMax ever after a lot of trying and scripting. Now I actually blame the hardware generation and specifically DDR3 for that, so with DDR4 and V3 processors you probably can overcome the memory BW limitations that I cant. However it also makes me unable to offer you practical advice on staggering, especially in windows. Figuring out your HW controller issue however (its not the drives, its the controller causing that issue in ubuntu) is something I would give a try at helping you with. It would also be your overall fastest plotting potential.
Title: Re: Dual plotting issues, one of the nodes underperforming
Post by: Creative on October 11, 2021, 02:53:26 PM
Yes, I´m really sad for not being able to use Ubuntu, I actually blame Windows for my times, not Madmax. Before this I kind of avoided Linux at all cost in my whole life, but with Chia I started to learn, and I even started to like it a lot.

My main fear is that with only 36 disks connected its giving me a lot of issues with the drives, I tried everything, flashed my LSI 9211 - 8i to IT mode with P20 firmware, tried different methods for formatting with Parted, gParted, fdisk, all of that and nothing, I cannot imagine what things will I encounter when I connect all of my 120+ drives, my concern is finding out halfway plotting that there is no way to connect lets say 40 percent of the drives since I want to buy connectivity hardware while I finish plotting the ones connected.

I really appreciate if you can shed some light on my controller issues. For background, I have the controller connected with two HPE - 487738-001 SAS expanders cards. I  actually keep Ubuntu in another SSD to dual boot.

Thank you so much again, If you´d like to connect in a more direct way through Discord or something please just let me know.

Kind regards!
Title: Re: Dual plotting issues, one of the nodes underperforming
Post by: Administrator on October 11, 2021, 04:14:25 PM
AH yes. Those expanders. Yeah  :-\ I considered those expanders back in 2013 before going with a SC846 setup. I remember passing on them for the following reason.

https://hardforum.com/threads/hp-sas-expander-owners-thread.1484614/

I take it you have read this thread on their known issues with certain drives and failure to initialize? To my knowledge they never got FW patched and are something I would not put into production.

They work in windows with the driver supporting the legacy mode operations, but much more reliable expanders can be had. I think if you had a really old kernel for linux (3.16?) you would see that same support, but some drives just don't play well with them.
Title: Re: Dual plotting issues, one of the nodes underperforming
Post by: Creative on October 11, 2021, 06:03:30 PM
Yes, I think the expanders are the one creating my issues because on windows some times it takes long to recognize the disks after restarting.
I've been many days trying to find a firmware update for the expanders and how to upgrade them but no luck, perhaps this would fix my issues.
I'm only connecting these disks this way, I found a better and cleaner solution using 24 bay backplanes from supermicro that should work much better.
Do you know of any link for the firmware and some explain on how to install?
Thanks!
Title: Re: Dual plotting issues, one of the nodes underperforming
Post by: Administrator on October 31, 2021, 04:06:49 PM
The didnt update their firmware ever is the issue  :'(
Title: Re: Dual plotting issues, one of the nodes underperforming
Post by: Creative on December 05, 2021, 01:22:38 PM
Hi,

Actually after a lot of testing, the issue are actually the controllers LSI 9211-8i that are not working well in my hp system under Ubuntu, in Windows they work, that´s the odd part. I have bought a Dell H200 a couple of days ago and I´m getting exactly the same issue, I really don´t know what controller o buy, I´m throwing a lot of money on useless stuff.
Both HP SAS expanders are working now with my internal P440ar but it is not capable of "daisy connection" between expanders, so I really need to get some other SAS controller working asap.
I belive there has to be some driver problem between LSI 9211-8i and Ubunut, I really don´t know what to think or do.

Hope you can help me, thanks in advance!

Regards.