Deduplication info not visible for volumes in storage manager

[ThomasGoering]

I successfully enabled btrfs data deduplication for my two hdd volumes with the script on my DS920+ running DSM 7.2.1-69057 Update 3. I use it for my two volumes that are on hdd-only storage pools and it seems that it is working fine.

But I don't see any deduplication info. I just see file system and total capacity. Deduplication status and savings are missing.

From what I found out so far in file storage_panel.js, it seems that this.dedup_info.show is always false. Even if it would be true it wouldn't display the correct status because this.dedup_info.status is "disable" (but it should be "auto" for two volumes). Unfortunately I have not found out where these values come from.

Is something missing on my DS920+? It would be great if the deduplication info can be displayed in storage manager.

[007revad]

I can't test it until tomorrow because my DS1821+ is doing a data scrub, and the drives from my DS720+ are currently connected to a PC for developing my Synology_Recover_Data script.

What we need to change may be in storage_panel.js or storage_wizard.js storage.js

[007revad]

I don't know which image from #25 you are referring to because I can't see your screenshot. All I get is a webpage saying "This private-user-images.githubusercontent.com page can’t be found".

[007revad]

Out the 43,000 lines in storage_panel.js only 371 line contain the word dedupe.

If we search for dedupe related functions there are only 23.

	Line  8992:                 dedupeConfigBtnShow() {
	Line  8996:                 dedupeConfigBtnEnable() {
	Line  9145:                 async configureDedupe() {
	Line 14158:             get validDedupeDisk() {
	Line 14468:             get showDedupInfo() {
	Line 14471:             get showDedupRate() {
	Line 14474:             get dedupStatusText() {
	Line 14489:             get dedupRateText() {
	Line 21546:                 checkAndAlertAnotherVolumeDeduping() {
	Line 21557:                 async btnStopDedupe() {
	Line 21573:                 async btnManualDedupe() {
	Line 21636:                 isManualDeduping() {
	Line 21901:                 checkAndAlertAnotherVolumeDeduping() {
	Line 28620:                 doVolumeDedupe() {
	Line 28623:                 showDeduplicationStep() {
	Line 28662:                 nextStepOfDedupe() {
	Line 36412:                 async openDedupScheWin() {
	Line 36436:                 openDedupHelp() {
	Line 36535:                 hasDedupEnabledVol() {
	Line 36538:                 dedupEnabledVolumeLabel() {
	Line 36542:                 dedupEnabledVolumeHtml() {
	Line 36547:                 dedupEnabledVolumeTip() {
	Line 36553:                 dedupEnabledDescCls() {

My javascript knowledge is very basic so I use ChatGPT to tell me what the function is doing 😄

get showDedupInfo() {
    return !!this.dedup_info && this.dedup_info.show
}

ChatGPT said:
"This JavaScript function showDedupInfo seems to be a getter function for a property dedup_info within some object. Here's what it does:

It checks if this.dedup_info exists and is truthy (!!this.dedup_info). The double negation !! is a common way to coerce any value to a Boolean.
If this.dedup_info is truthy, it then checks if this.dedup_info.show is also truthy.
It returns true if both conditions are met (both this.dedup_info and this.dedup_info.show are truthy), otherwise it returns false.
So, essentially, this function is checking whether dedup_info exists and if its show property is also truthy, returning true if both conditions are met, and false otherwise."

So I would try:

get showDedupInfo() {
     return !!this.dedup_info
}

Or this:

get showDedupInfo() {
     return this.dedup_info.show
}

Or even this:

get showDedupInfo() {
     return true
}

showDedupRate may also play a part in what is shown or when:

get showDedupRate() {
    return !!this.dedup_info && "auto" === this.dedup_info.status
}

[ThomasGoering]

I don't know which image from #25 you are referring to because I can't see your screenshot. All I get is a webpage saying "This private-user-images.githubusercontent.com page can’t be found".

I'm sorry, the link included my session data I assume. Here's the screenshot I was referring to:

[ThomasGoering]

In the meantime I looked deeper into storage_panel.js and I doubt that the deduplication info can fully be enabled just by changing the JavaScript. It's like with "&&e.dedup_info.show_config_btn" that you removed to enable hdd support.

debug_info get its values from a web service call:

                       e.result.forEach((e => {
                            if (e.success)
                                switch (e.api) {
                                case "SYNO.Storage.CGI.Storage":
                                    ({missing_pools: s, detected_pools: i, disks: a, env: o, ssdCaches: r, sharedCaches: n, storagePools: l, volumes: d, overview_data: u} = e.data),
                                    e.data.AHAInfo && (o.AHAInfo = e.data.AHAInfo);
                                    break;

Then d is an array (of volumes). In my case it contains (in reverse order!) my three volumes (one ssd volume and the two hdd volumes mentioned above) and e.g. d[2].dedup_info looks like this (this is one of my hdd volumes - in fact it is volume1 - that has data deduplication enabled and already ran it manually and automatic at the moment as well):

btn_alert: ""
last_run_time: 0
show: false
show_config_btn: false
status: "disable"
total: 0
used: 0

So all these values indicate that data deduplication is not active for this volume but it is! This might be because it's an hdd volume.

Only for my ssd volume dedup_info.show_config_btn is true. That's why you removed the check for it: To support data deduplication for hdd volumes.

With that being said, in my opinion it doesn't help to remove or change checks for dedup_info.show or dedup_info.status. Even if this would display something it wouldn't display the correct values, because e.g. the real savings are not in dedup_info.

To fix all this it must be fixed in the web service implementation on the DS. Is it known if this is something that can be achieved?

[007revad]

What are we looking at in that screenshot? Is something missing?

[ThomasGoering]

What are we looking at in that screenshot? Is something missing?

Nothing is missing in the screenshot, which I took from issue #25 just to show what is missing in my Storage Manager.

In the screenshot there's the expanded volume, and its "Info" includes "Deduplication status:" and "Deduplication savings:".

These two are missing for my dedupe-enabled hdd volumes. I only see "File system:" and "Total capacity:".

[007revad]

In the screenshot there's the expanded volume, and its "Info" includes "Deduplication status:" and "Deduplication savings:"

Ah, that's what the screenshots in issue #25 were showing. I only noticed the choices in the ... menu. I didn't notice that the "Deduplication status:" and "Deduplication savings:" were missing under the "Info" section. I don't think I've seen them before. I never realised something was missing.

I can't find SYNO.Storage.CGI.Storage in DSM 7.2.1 with Update 1 or in Storage Manager 1.0.0-0017

Storage Manager 1.0.0-0017 does have /usr/local/packages/@appstore/StorageManager/webapi/SYNO.Storage.CGI.lib

And for DSM 7.2 where Storage Manager was part of DSM it's /usr/syno/synoman/webman/modules/StorageManager/SYNO.Storage.CGI.lib

[ThomasGoering]

My understanding so far is: SYNO.Storage.CGI.Storage is "defined" in SYNO.Storage.CGI.lib (which is JSON). It is an API and has a method load_info. If it is called it returns a JSON that includes - among all the other storage pool and volume information - the dedup_info object for each volume.

libStorage.so also contains load_info but then I don't know. I haven't found dedup_info in a file yet on the DS (other than in storage_panel.js). And if found it might be the JSON renderer which wouldn't probably help a lot. But I'll keep digging...

This is another way of calling load_info:

sudo /usr/syno/bin/synowebapi --exec api=SYNO.Storage.CGI.Storage method=load_info version=1

There is another API called when you click "Configure Data Deduplication" in Storage Manager. Using synowebapi and two volumes - instead of one volume when called from Storage Manager - it would be:

sudo /usr/syno/bin/synowebapi --exec api=SYNO.Storage.CGI.BtrfsDedupe method=get_volume_info version=1 vol_paths=[\"/volume1\",\"/volume2\"]

It is strange that deduplication works in general but load_info doesn't seem to be aware of it...

Last update here:

I found dedup_info in /lib/libsynostoragemgmt.so. It contains symbol SYNOBtrfsDedupeAllowedSSDModelsGet, so it seems it does something similar in the web service than the Storage Manager does in the browser (see issue #46). I had a quick look at the file with IDA free (first time using it) but don't know if it is implemented in this lib or it is called in another one. Here my capabilities are limited but maybe it helps whoever wants to dig into it...

[007revad]

I just played around with deduplication on an NVMe volume.

For manual deduplication "Deduplication status" shows the last run time and there is no "Deduplication savings".

For automatic deduplication "Deduplication status" shows Automatic and "Deduplication savings" shows either:

• "Waiting to run data deduplication" if auto dedupe has not run yet.
• The bytes/KB/MB/GB/TB of space saved.

For the NVMe volume with automatic deduplication enabled but not run yet debug_info is:

"btn_alert" : ""
"last_run_time" : 0
"show" : true
"show_config_btn" : true
"status" : "auto"
"total" : 0
"used" : 0

For the same NVMe volume with automatic deduplication disabled, and after manually running deduplication debug_info is:

"btn_alert" : "",
"last_run_time" : 1714520572,
"show" : true,
"show_config_btn" : true,
"status" : "disable",
"total" : 0,
"used" : 0

It sounds like I need to edit /lib/libsynostoragemgmt.so so that SYNOBtrfsDedupeAllowedSSDModelsGet returns true.

[007revad]

When I run the following command with just HDD volumes in the NAS:

/usr/syno/bin/synowebapi --exec api=SYNO.Storage.CGI.Storage method=load_info version=1

the first line is always:

[Line 295] Exec WebAPI:  api=SYNO.Storage.CGI.Storage, version=1, method=load_info, param={}, runner=SYSTEM_ADMIN

But when I run that command with an SSD volume in the NAS there was no [Line 295] error.

SYNO.Storage.CGI.lib contains a single line, so I'm not sure if it's a synowebapi error or not.

[ThomasGoering]

Maybe it's not indicating an error but just echoing what it's doing?

I have two HDD volumes and one SSD volume in my DS920+ (and DX517) and get [Line 295] Exec WebAPI... in the first line of the output from both synowebapi calls that I mentioned in my last comment.

[007revad]

The results from googling for [Line 295] +"Exec WebAPI" seem to suggest that [Line 295] Exec WebAPI... is normal. It's just strange that it was missing after I created an SSD volume. Though I am getting that first line today even thought I've changed nothing (I still have 2 HDD volumes and an NVMe volume).

[cb12tre]

I also tried creating two volumes with deduplication, one on an HDD and one on an NVMe. However, deduplication info is not displayed for the HDDs. The system is a 920+ running DSM 7.2.1-69057 Update 5, with Storage Manager 1.0.0-0017.

While not essential for functionality, having this information would be very convenient.

[cb12tre]

I was wondering if it's possible to view deduplication statistics via SSH since they aren't directly visible in the graphical interface?

[ChengYen-Tang]

Hi @007revad,
Why, I don't see any deduplication information

[ChengYen-Tang]

By the way, my deduplication rate feels pathetic.😥
My ISO folder contains a large number of Windows, Windows Server, Office ISOs. In theory, this should have a high deduplication rate.

[007revad]

@ChengYen-Tang

Why, I don't see any deduplication information

I haven't figured out how to show "Deduplication status" and "Deduplication savings" in the volume's info panel for HDDs.