Seagate ST2000NM00XX Drive Various Power Management Not Supported

[andersonsmith]

Hello, thanks for the work on this toolset. I have a system running 8 seagate SAS drives and cannot get spindown to work. I first worked with sdparm and would see it bouncing in and out of idle. I was hoping to use this to monitor the timer and see if it's getting passed idle_a. The drive manuals for the three models I have all say they support PowerChoice and have the 3 idle states as well as 2 standby states. When polling the drives, I get the following message returned.

==========================================================================================
 openSeaChest_PowerControl - openSeaChest drive utilities - NVMe Enabled
 Copyright (c) 2014-2024 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
 openSeaChest_PowerControl Version: 3.6.0-8_0_1 X86_64
 Build Date: Aug 28 2024
 Today: 20240902T091153 User: root
==========================================================================================

/dev/sg7 - ST2000NM0001 - Z1P20M8C - XRBA - UNKNOWN
Showing EPC Settings not supported on this device.

I have verified that EPC is enabled, so I'm not sure exactly what to do next.

       Features Supported:
                Protection Type 1
                Protection Type 2
                Application Client Logging
                Self Test
                Automatic Write Reassignment [Enabled]
                Automatic Read Reassignment [Enabled]
                EPC [Enabled]
                Informational Exceptions [Mode 6]
                Background Media Scan [Enabled]
                Translate Address
                Format Unit
        Adapter Information:
                Adapter Type: PCI
                Vendor ID: 1000h
                Product ID: 0097h
                Revision: 0002h

For reference, the drives are all connected through a SAS3008 HBA flashed to IT mode. The HBA is integrated into a Gigabyte MD70 motherboard. The host is Proxmox running ZFS on the drives.

Please let me know if I've missed anything. Thanks again.

[vonericsen]

Hi @andersonsmith,

I think the reason you are getting this message is because it did not detect the device as a SCSI drive type (It is saying UNKNOWN in your output).
I'm not sure why this is not detecting as a SCSI drive type, but this seems to be a software bug so I will debug it to figure out where it went wrong since the rest of the output makes sense.
This is what is also preventing displaying the EPC timer values since it is checking the drive type to figure out how to read it.

An alternative to --spindown is --transitionPower standby or more specifically for EPC --transitionPower standby_z (these should be the same though).
Can you let me know if this works?
This would be good to know as the code paths between spindown and transition power are a little different and it may be better to make them the same if transition power works correctly.

The other thing I think can help me figure out where this bug is coming from is the output from openSeachest_PowerControl -d <handle> --llInfo
That is two lowercase Ls and a capital I for info (short for low-level info).
This will dump a lot of information that openSeaChest reads from /sys to figure out how the drive is attached, the adapter IDs (which read correctly for your output), and some other information that may help me figure out what went wrong and triggered this bug.

[vonericsen]

Oh, one more thing you can do to workaround the UNKNOWN drive issue is to use the --forceSCSI option which will override the drive type detection (or in this case miss-detection) which should allow it to read and dump the EPC settings.

[andersonsmith]

Thanks for the replies. Using --forceSCSI did indeed allow the timers to show correctly. I tried to force the standby mode with --transitionPower standby_z but only received Device is in the Active State or in an Unknown state. afterwards. I continued to every 5 seconds or so but it didn't change from this.

Regarding the info dump, here it is:

==========================================================================================
 openSeaChest_PowerControl - openSeaChest drive utilities - NVMe Enabled
 Copyright (c) 2014-2024 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
 openSeaChest_PowerControl Version: 3.6.0-8_0_1 X86_64
 Build Date: Aug 28 2024
 Today: 20240903T205432 User: root
==========================================================================================

/dev/sg7 - ST2000NM0001 - Z1P20M8C - XRBA - UNKNOWN

---Low Level tDevice information---
        ---Drive Info---
                media type: HDD
                drive type: unknown
                interface type: SCSI/SAS/FC/etc
                zoned type: not zoned
                ---adapter info---
                        PCI/PCIe:
                        VendorID: 1000h
                        ProductID: 0097h
                        Revision: 0002h
                ---driver info---
                        driver name: mpt3sas
                        driver version string:
                SCSI Version: 6
                ---Passthrough Hacks---
                        Passthrough type: SAT/system/none
                                ---SCSI Hacks---
                                ---NVMe Hacks---
                                ---ATA Hacks---
        ---OS Info---
                handle name: /dev/sg7
                friendly name: sg7
                minimum memory alignment: 8
                ---Linux Unique info---
                        FD is valid
                        SG Driver Version:
                                Major: 3
                                Minor: 5
                                Revision: 36
                OS read-write recommended: false
                last recorded error: 0
                File system Info:
                        No active file systems detected

[vonericsen]

@andersonsmith,

This bug has been very difficult to locate so far walking through the code with the information you have shared so far.
Do you see this message show up at all? SCSI interface, unknown drive type

Can you run this command line and share the output with me?
openSeaChest_PowerControl -d <handle> --transitionPower standby_z --checkPowerMode -v4 | tee verboseSCSIPower.txt

This will run with a lot of verbose output showing me all the command issued and all the data transfers to and from the drive for me to validate manually to see if something else is going on unexpectedly.

[andersonsmith]

Where would I find that error message?

I ran the command with and without --forceSCSI in case that matters.

verboseSCSIPower.txt
verboseSCSIPower-withforce.txt

[vonericsen]

Thank you for the logs!
I believe I have identified the issue causing "Unknown Drive" so that --forceSCSI is not necessary.

Looking through the output with --forceSCSI I see the command that is issued (request sense) to check the power state and it is just reporting a normal "good" status.
While reviewing the standards I can try adding another command into this process to see if it can detect this state: test unit ready.
It may not give as much detail, but I think I can make that work.

You can try spinning the drive down, then trying openSeaChest_PowerControl -d <handle> --testUnitReady to see if it has spun down. If it has spun down, then the sense will most likely say Logical Unit Not Ready - Initializing Command Required which essentially means it is in standby.
I'll work on adding this in as well, but if you could try this and let me know what you see that would also be helpful to confirm what the drive is doing and reporting in your case.

[vonericsen]

@andersonsmith,

Can you give the attached build a try and let me know if it works for you to show standby?
If it does not, can you share the output again like you did previously?
openSeaChest_PowerControl -d <handle> --transitionPower standby --checkPowerMode -v4 | tee verbosePowerCheck.txt

linux-x86_64-portable.zip
multiple.intoto.jsonl.zip

[andersonsmith]

Output attached. When I check with --showEPCSettings immediately after it went back to Active or unknown state but I assume that due to disk activity.

verbosePowerCheck.txt

Should an asterisk be shown with each timer in --showEPCSettings? I see it appearing and disappearing when I enable and disable idle_a but it doesn't appear when I enable standby_y or standby_z

[vonericsen]

@andersonsmith,

Thanks for the log again!

When I check with --showEPCSettings immediately after it went back to Active or unknown state but I assume that due to disk activity.

Yeah, even querying the settings can potentially spin up a disk. It depends if the disk stores the settings on the media or in flash or RAM.
This can change between products as well.

It could also be some other system activity (even something you are not triggering directly, but the OS decides it needs something from the device).

One other thing I have observed in the past is depending on how the drive's handle is opened in Linux, that can cause the drive to spin up.
When I had this reported years ago the issue was using the sd? handle caused a spinup, but using sg did not.
My solution to this to keep customer usage easy for the handle they are familiar with was to map it myself using sysfs.
If the mapping fails, it uses whatever handle the user provides since some kernels disable /dev/sg? handles (usually only embedded devices).
This issue is not appearing in your logs since I see it using the sg handle, but it is always possible that something in the kernel has changed this behavior, although I am not aware of anything that would affect this right now.

One other thing that may factor into how this is working is the drive firmware. In your case the firmware string I see indicates a customer unique firmware and sometime those have customizations that may cause some changes from the default behavior of OEM code. I don't know of any specific changes to power settings on this code, but I have not worked with it a lot. I think most likely it is spinning back up due to querying the mode page for the current settings or some other background activity from the OS.

SAS drives also have one field that allows specifying preference between background tasks and power modes which could be another part of it. openSeaChest_PowerControl does not currently have a way to toggle this, but I can look into this as part of troubleshooting this issue. We can also try flipping it using openSeaChest_Configure's options to change a mode page with some raw field info if needed.

Should an asterisk be shown with each timer in --showEPCSettings? I see it appearing and disappearing when I enable and disable idle_a but it doesn't appear when I enable standby_y or standby_z

Yes, that should be changing.
I just reviewed the code and this is a bug.
It looks like when reading one page there was a mix up on which bit was used for standby y vs standby z. I also found a bug reading the mode page settings where the length of the mode page header was not being factored into the offset to check.
This only seems to affect the standby states which is why you didn't see the issue with idle a.

I'm reviewing more of the code before I push the fix, but I will share that for you to check out as well.

[vonericsen]

@andersonsmith,

Can you give these updated builds a test to see if it fixes the asterisk in the EPC settings?
I fixed the issues I mentioned in my last comment that should resolve it.

multiple.intoto.jsonl.zip
linux-x86_64-portable.zip

[andersonsmith]

Thanks for the deep dive on this one! Can confirm that the asterisk is showing in the EPC settings output now. I assume you are referring to this output from sdparm when referring to the bias for power management?

   /dev/sg7: SEAGATE   ST2000NM0001      XRBA
   Direct access device specific parameters: WP=0  DPOFUA=1
Power condition [po] mode page:
 PM_BG         0  [cha: n, def:  0, sav:  0]  Power management, background functions, precedence
 STANDBY_Y     1  [cha: y, def:  0, sav:  1]  Standby_y timer enable
 IDLE_C        0  [cha: n, def:  0, sav:  0]  Idle_c timer enable
 IDLE_B        1  [cha: y, def:  0, sav:  1]  Idle_b timer enable
 IDLE_A        1  [cha: y, def:  0, sav:  1]  Idle_a timer enable
 STANDBY_Z     1  [cha: y, def:  0, sav:  1]  Standby_z timer enable
 IACT          10  [cha: y, def: 10, sav: 10]  Idle_a condition timer (100 ms)
 SZCT          9000  [cha: y, def:36000, sav:9000]  Standby_z condition timer (100 ms)
 IBCT          2400  [cha: y, def:6000, sav:2400]  Idle_b condition timer (100 ms)
 ICCT          0  [cha: n, def:  0, sav:  0]  Idle_c condition timer (100 ms)
 SYCT          6000  [cha: y, def:18000, sav:6000]  Standby_y condition timer (100 ms)
 CCF_IDLE      0  [cha: n, def:  0, sav:  0]  check condition if from idle_c
 CCF_STAND     0  [cha: n, def:  0, sav:  0]  check condition if from a standby
 CCF_STOPP     0  [cha: n, def:  0, sav:  0]  check condition if from stopped

Specifically thinking of the top line? I assume that it is currently set to power management rather than background.

Also figured that the firmware is because these drives are Dell drives so appear to be specific firmware. Some settings seem to be locked such as the background media scan. Does not appear to be editable. Same with the power management bias.

[vonericsen]

Thanks for reporting that fixed the asterisk!

I assume you are referring to this output from sdparm when referring to the bias for power management?
Specifically thinking of the top line? I assume that it is currently set to power management rather than background.

Yes, the PM_BG line is what I'm referring to in this case.
When set to zero like in your output this is defined as vendor specific in the spec, which is used to say it's up to the firmware to decide this precedence between background tasks and power management. This is what the defined behavior of SATA drives is as well since there is no method to change this behavior on SATA.
Sometimes this also is a "backwards compatibility" setting when a field used to be unused, it means it was not defined in a previous standard or draft of a standard and in newer standards it now has meaning, so previous behavior of that field was left up to the firmware.

A firmware may also set something to vendor specific when it was incapable of changing that behavior. That may be selectable on some drive firmware, but I do not know off the top of my head.

Also figured that the firmware is because these drives are Dell drives so appear to be specific firmware. Some settings seem to be locked such as the background media scan. Does not appear to be editable. Same with the power management bias.

This could be the customer specific firmware or just that the firmware at that time did not have a way to support changing that field. I don't know what the STDOEM firmware would do on this drive so I cannot confirm in this case.

One other thing I think you may find helpful for figuring out if your drive is going into these low power modes or not is that in openSeaChest_SMART there is the option --deviceStatistics which can print out some counters for how many times a drive has entered these different power states. I cannot say for sure that it is supported on this firmware or not, but you can try that and check the counter to see if it's incrementing over time or not.

If you think it would be helpful to put the PM_BG output into openSeaChest, let me know and I will look into that too.

[andersonsmith]

Thanks for the suggestion. Looks to me like it's having trouble getting beyond idle_A The timer for idle_B is set to 4 min and standby_Z is set to 15 minutes for this particular drive. Will dive into the stats to see what the system is doing. This is mostly media so should spend a significant amount of the day idle. Thanks again for your work on this - this is a very handy tool.

---Power Condition Transitions---
 Accumulated Transitions to Active                            N/A              589
 Accumulated Transitions to Idle A                            N/A              4431195
 Accumulated Transitions to Idle B                            N/A              583
 Accumulated Transitions to Idle C                            N/A              0
 Accumulated Transitions to Standby Z                         N/A              6
 Accumulated Transitions to Standby Y                         N/A              0

[vonericsen]

@andersonsmith,

No problem! Thanks for reporting this issue.

I'm currently working on writing up Wiki pages to not only help with openSeaChest use but also to help educate on how drives and their features work.
I have not yet started a page on power options like EPC, but I will put some of the information from here into that wiki page when I get to writing it.

[vonericsen]

I have made a patch release v24.08.1 that fixes changing EPC settings and the SCSI drive detection.

edit: My earlier comment was about another issue I missposted so you can ignore that 😄