Fabric missing costs after shortcut to ingestion

[lundejd]

I'm getting things moved to Fabric and am curious.. I believe it was .6 that changed the file path around to better fit Fabric, does that mean I should move some of the older folders? I'm curious if this is why I'm missing costs in Fabric vs. direct with Power BI desktop.

I have a shortcut to ingestion, but inside it has both focus cost and providers. I can even see a couple .parquet files in October's data, then 8 of them in the new folder structure. (seems like duplicate data but I'm not seeing more I'm seeing less)

For September I'm seeing 15 parquet files in the old structure, and 15 in the old. I don't have any August or earlier in the new folder structure.

I realize Fabric probably isn't completely supported yet but I wanted to get a jump and at least get to the point where my data was consistent between the two. It's the same exact data, but does the Dataflow Gen2 understand to use the entire path and both folder structures? (I have only loaded 2 months of data thus far because that alone still takes the dataflow 4+ hours)

There are other strange things like I can't move between number of months to a date range without the load failing, but for now I just need to get the folder structure and data consistent.

My configuration is a shortcut to the ADLS as mentioned, a Gen2 Dataflow and a LakeHouse. Copy/pasted everything from the CostDetails.pbix file into the Dataflow so it looks identical to my Power BI file. I'm going against a FinOps setup that was installed new with .5, backfilled a year, then upgraded to .6 which changed the folders.

Type = folder, relative path = files/ingestion, type = AdlsGen2, Shortcut target subpath = /ingestion.

Loving what I see in fabric. A game changer, especially for a team that shares all this data. I won't go back.

[lundejd]

Fixing my own issue - looking at the code, it was ftk_Storage that couldn't process the new path because the url given from the one lake is 3 "/" characters longer than the one we present for ADLS. Adding 3 in a couple places fixed my issue.

The reason it was picking the old path was that it could still take the longer url and do an n-1 to find the old file path date so it was going that route.

[flanakin]

0.6 moved the dataset to be the first folder under the container. The reference to Fabric with this change was due to shortcuts and notebook references so you can have a single folder that contains all data for that dataset. This is not explicitly required, but it makes things easier. If you're only using cost data, this isn't needed, tho. It's mainly to support multiple datasets.

If you want to clean up storage, you can either move the files around to match the new structure or re-export data to have it processed via 0.6 pipelines.

FYI - I would like to add a pipeline for "upgrading" data as well but haven't had time to do that yet. There's a new config_InitializeHub pipeline coming in 0.7 to support Azure Data Explorer setup but it doesn't have storage cleanup yet.

Glad to hear Fabric is working well. We'll get there. We have a few people looking at that now. The biggest challenge is figuring out how to reuse code across both. I'd love to hear how things are going.

Side note: Would it make sense to have a community call to discuss any issues you're running into when customizing things?

[lundejd]

Fabric has been a fun challenge. We have almost 13M rows in a billing month and fabric seems less friendly to working with a lot of data (which feels opposite of what it should be). To get around it we're having to work with a month at a time and do the processing in multiple steps via pipelines but it's working. It does seem odd to me that fabric can't process as much data and is much slower than refreshing a .pbix file, which I always thought was painfully slow to begin with.

Speaking of re-using code, I think there is a solution at least for those using LakeHouse and either a pipeline to pull in data or a shortcut pointing to their existing ADLS:

One difference between Fabric and the new folder structure is the depth. The current ftk_Storage code has the following:


This works when your URL =

In Fabric, the URL looks like this:

I placed my ADLS shortcut “ingestion” right under the “Files” folder, but someone else might have sub-folders under Files which would add yet another layer per. To ultimately solve and be able to reuse code between fabric or ADLS I’d think you’d want to do the same List.RemoveLastN tactic because the ending of both should always be the same regardless of where you might have them - in an Azure ADLS, or in Fabric Lakehouse.

Something like:

 // Supports 0.6+ folder structure: {dataset}/{yyyy}/{MM}/{scope} in both ADLS and Fabric
                    if Text.Length(_toMonth(List.Last(List.RemoveLastN(List.RemoveLastN(Text.Split([Folder Path],"/"), 1), 5)))) = 4 and TextLength(_toMonth(List.Last(List.RemoveLastN(List.RemoveLastN(Text.Split([Folder Path],"/"), 1), 4)))) = 2 then
                        _toMonth(List.Last(List.RemoveLastN(List.RemoveLastN(Text.Split([Folder Path],"/"), 1), 4)))
                    // Supports 0.2-0.5 folder structure: {scope}/{yyyyMM}/{dataset}
                    else  _toMonth(List.Last(List.RemoveLastN(List.RemoveLastN(Text.Split([Folder Path],"/"), 1), 1)))
                otherwise _toMonth(List.Last(List.RemoveLastN(List.RemoveLastN(Text.Split([Folder Path],"/"), 1), 1))),
                type date

I have not tested the above code but will get around too it but you get the idea.

[flanakin]

I love this! Adding support for the OneLake path is an easy change. Is this with the ADLS connector or the OneLake connector?

[lundejd]

I first create an ADLS shortcut in a Lakehouse used for the FinOps toolkit, then use the shortcut URL for dataflows.