character or factor in dataset

[clarkliming]

in rtables, both character and factor are allowed, however there are cases that lead to the following error like

Error: Error applying analysis function (var - RACE): Number of rows generated by analysis function do not match across all columns.

This error message is not clear to the end users.
So the question here is: shall chevron always require "factor" used, or allow "character"?

option 1: chevron require factors be used. And in preprocessing convert all necessary variables to factor.
pro:

1. easier to use (minimal requirement for the data)

cons:

1. preprocessing is much complicated
2. custom preprocessing require more user effort

option 2: chevron require factors/character be used. In preprocessing convert some of the variables to factor (only if called with analyze_vars. Any other possibilities that this error could occur? @Melkiades )
pro:

1. easier to use (minimal requirement for the data)
2. minimal changes needed

cons:

1. only several preprocessing is complicated

option 3: chevron still allows characters (as this is supported by rtables), and rtables/tern may improve the handling of such situation

pros:

1. users are able to understand the error even if they are not using chevron
2. chevron will still be simple (and robust to data) from the users' side (in most of the cases; only in cases that different arm have different levels will this issue occur)

cons:

1. users still need to manually adjust the data (convert to factor) if rtable/tern only provide "warnings" if not provide solution to make it work
2. to eliminate the error can be incompatible with the current design

@Melkiades may I ask if there are any differences between factor and character in rtables?

[clarkliming]

or @Melkiades do you think if it is possible to allow the different length in results? use "" as the place holders should be fine?

[clarkliming]

related to #675

[barnett11]

Just bumping this again @clarkliming - we really need to resolve this before more users start onboarding into this space as it's quite a fundamental

[BFalquet]

I think Admiral/Oak should do it. At our stage, converting to factor will drop the missing levels and it will be difficult to track.

[clarkliming]

how about we only allow "factor" ? @BFalquet maybe you can have a look and summarize

1. how many variables are allowing factor/charactor only, or both
2. what impact we will have if we only allow factors

[Melkiades]

I think rtables transforms everything into factors, which is why we do df_explicit_na. Otherwise, that will be eaten up by the transformation. I have been thinking to update how we deal with NAs for a bit, but still haven't got the time to work pro-actively on that

[Melkiades]

btw sorry for missing this out. The error seems unrelated to the discussion about factor vs character. I need to see why it happens but it seems that some analyze outputs more or less statistics than others. Looking at the error, I think this is not allowed if not with pruning or multiple tables' rbind. I need to take a closer look at the code producing the error

[clarkliming]

btw sorry for missing this out. The error seems unrelated to the discussion about factor vs character. I need to see why it happens but it seems that some analyze outputs more or less statistics than others. Looking at the error, I think this is not allowed if not with pruning or multiple tables' rbind. I need to take a closer look at the code producing the error

thank you @Melkiades there are multiple reasons of this issue, but "character" can lead to this. there is even an extra intro like "converting something to factor". it happens when in each column the unique character values do not match and we want to have a summary table counting the occurence of each level.