Exporting metrics

[llrs-roche]

PR to address #376

Challenges:

1. exporting metrics directly to PACAKGES results on floating numbers with different accuracy (on a test case between 17 and 19 digits)
2. To recover the same object (aka identical(metrics, metrics |> write() |> read())) means restoring the labels and some attributes of the metrics:

attributes(metrics$downloads_1yr)
## $.recording
## $.recording$expr
## {
##     sum(x$downloads$count)
## }
## 
## $.recording$attributes
## NULL
## 
## $.recording$visible
## [1] TRUE
## 
## ...

[rabiibouhestine]

@dgkf @llrs-roche Following our discussion last Friday, I wrote a function pkg_metric_export() that exports scored package metrics into a flat list of character strings in DCF. It supports both output classes of pkg_score() (list or tbl_df).

An example output for a single package:

testOne <- pkg_ref(utils) %>%
  pkg_assess() %>%
  pkg_score() %>%
  pkg_metric_export()

MetricPkgScore: 0.6870599
MetricCovrCoverage: NA
MetricHasNews: 0
MetricRemoteChecks: NA
MetricNewsCurrent: 0
MetricRCmdCheck: NA
MetricExportedNamespace: 0.07135333
MetricHasVignettes: 1
MetricExportHelp: 1
MetricHasWebsite: 0
MetricHasMaintainer: 1
MetricBugsStatus: NA
MetricSizeCodebase: NA
MetricHasSourceControl: 0
MetricHasBugReportsUrl: 0
MetricDownloads1yr: 0.00000
MetricReverseDependencies: 1
MetricHasExamples: 0.9937107
MetricDependencies: 0.88079708
MetricLicense: NA

An example output for multiple packages:

testMany <- pkg_ref(c("utils", "dplyr")) %>%
  pkg_assess() %>%
  pkg_score() %>%
  pkg_metric_export()

Package: utils
Version: 4.4.3
MetricPkgScore: 0.6870599
MetricCovrCoverage: NA
MetricHasNews: 0
MetricRemoteChecks: NA
MetricNewsCurrent: 0
MetricRCmdCheck: NA
MetricExportedNamespace: 0.07135333
MetricHasVignettes: 1
MetricExportHelp: 1
MetricHasWebsite: 0
MetricHasMaintainer: 1
MetricBugsStatus: NA
MetricSizeCodebase: NA
MetricHasSourceControl: 0
MetricHasBugReportsUrl: 0
MetricDownloads1yr: 0.00000
MetricReverseDependencies: 1
MetricHasExamples: 0.9937107
MetricDependencies: 0.88079708
MetricLicense: NA

Package: dplyr
Version: 1.1.4
MetricPkgScore: 0.3988960
MetricCovrCoverage: NA
MetricHasNews: 1
MetricRemoteChecks: NA
MetricNewsCurrent: 1
MetricRCmdCheck: NA
MetricExportedNamespace: 0.04611914
MetricHasVignettes: 1
MetricExportHelp: 1
MetricHasWebsite: 1
MetricHasMaintainer: 1
MetricBugsStatus: 0.4333333
MetricSizeCodebase: NA
MetricHasSourceControl: 1
MetricHasBugReportsUrl: 1
MetricDownloads1yr: 0.99194
MetricReverseDependencies: 1
MetricHasExamples: 0.9385965
MetricDependencies: 0.01098694
MetricLicense: NA

I think having a dedicated separate function pkg_metric_export() allows for more flexibility and easier control over its output without breaking backward compatibility for any tools currently using the package.

This is a starting draft version that basically only formats the current pkg_score() output and exports it as dcf. WDYT?

@dgkf @llrs-roche at this point does this qualify as a minimal example of what we're aiming for? should the values be score or more like assessments?

for example should we have this:

MetricHasWebsite: 1

or this :

MetricHasWebsite: www.websiteOne.com, www.websiteTwo.com

[dgkf]

This is looking great. I think we could use this as a target for the report to leverage as an input.

for example should we have this:

MetricHasWebsite: 1

or this :

MetricHasWebsite: www.websiteOne.com, www.websiteTwo.com

I'm not sure what the best path is here. My first impulse is that the exact urls are already in the DESCRIPTION, so maybe we just want them collapsed into a number for easier programmatic use.

[llrs-roche]

Sorry it took me some time to check on this. This looks promising.

When writing two metrics I found that it writes:

Package: utils
Version: 4.5.0
MetricDownloads1yr: 0.0000000

Package: dplyr
Version: 1.1.4
MetricDownloads1yr: 0.9919631

Note that instead of 0 as it is not an integer due to the dplyr call we get all these decimals. Do you think this could be modified? Or this is write.dcf fault (there are ongoing upgrades on write.dcf, so it might be a good time to ask for this feature iff needed) ?

I too agree that the number is better. If this is merged with the output of tools::write_PACAKGES the websites and other urls might already be there (Depending on the fields argument used [I have already asked that R by default it could include all fields but didn't receive any positive feedback]).

The function is not documented (and it takes a long time to document it given the complex documentation of the "double dispath" mechanism with roxygen2 7.3.2). But after documenting it worked well.

I think it would also make sense to explore how to read that file back into a score object (I would say even if it loses some attributes, as this could actually be a sign of the data origin)

[eddelbuettel]

@llrs-roche As you flagged this: we do not plan to touch write.dcf in the upcoming change to the DCF interface. So it looks like you may need to round / truncate / format ... before writing.

[llrs-roche]

Thanks @eddelbuettel for the info, we'll explore how we can approach this. Maybe just formatting each record and append them to a single file will work.

[eddelbuettel]

Sounds good. Also happy to chat more (over on that issue). Other than ensuring utf-8 (which is now more or less a given on all R platforms) we feel we have no TODO for write.dcf. Could this formatting issue be 'bubbled up the stack' to the functionality computing the metric?

[llrs-roche]

Was playing a bit more with the numeric precision problem:
When exporting just one package instead of 8 decimals we get 15!
MetricDependencies: 0.880797077977882 vs MetricDependencies: 0.88079708
This might be controlled by options(digits) but I am not sure as I don't think I've changed the options between the two runs and I don't see a way to specify as part of write.dcf call.

If we export just one package via: write.dcf(testMany[1, ], file = "out.txt") the MetricDownloads1yr is a double (with the 8 decimals), if we instead use a single one write.dcf(testSingle, file = "utils.txt") we get the integer: MetricDownloads1yr: 0.

We can work around this issue if instead of just as.character we use prettyNum which is precisely for converting numbers to nice characters: ops <- options(digits = 8);prettyNum(x$downloads_1yr);options(ops), this will also help controlling the decimals used when exporting just one or multiple metrics, and then to read back the object:

rd <- read.dcf("file.txt")
rd[, -c(1:2)] <- as.numeric(rd[, -c(1:2)])
rd

---

While exploring the inner works I realized pkg_ref is deleted instead of keeping the key information: the source where this has come from. It would be great if the score could keep that.
There might be some metrics that change depending on the system, for example some CRAN packages have some alias/help pages that are OS specific...

[rabiibouhestine]

@llrs-roche the numeric precision problem was caused by apply() converting each row of the data frame to a character vector. apply() coerces a data.frame into a matrix before applying the function. This coercion implicitly formats numeric values with lower precision (often 7–8 digits). I have pushed a fix for this.

As for pkg_ref I removed it because it was not a "metric" but an output of pkg_ref|() that I assumed not meant to be exported to PACKAGES file. Moreover, when running pkg_score() on a single package, the output does not have anything related to pkg_ref as far as I understood:

> pkg_ref("utils") %>%
  pkg_assess() %>%
  pkg_score() %>%
  names()
 [1] "covr_coverage"        "has_news"             "remote_checks"       
 [4] "news_current"         "r_cmd_check"          "exported_namespace"  
 [7] "has_vignettes"        "export_help"          "has_website"         
[10] "has_maintainer"       "bugs_status"          "size_codebase"       
[13] "has_source_control"   "has_bug_reports_url"  "downloads_1yr"       
[16] "reverse_dependencies" "has_examples"         "dependencies"        
[19] "license" 

> pkg_ref(c("utils", "dplyr")) %>%
  pkg_assess() %>%
  pkg_score() %>%
  names()
 [1] "package"              "version"              "pkg_ref"             
 [4] "pkg_score"            "covr_coverage"        "has_news"            
 [7] "remote_checks"        "news_current"         "r_cmd_check"         
[10] "exported_namespace"   "has_vignettes"        "export_help"         
[13] "has_website"          "has_maintainer"       "bugs_status"         
[16] "size_codebase"        "has_source_control"   "has_bug_reports_url" 
[19] "downloads_1yr"        "reverse_dependencies" "has_examples"        
[22] "dependencies"         "license" 

[llrs-roche]

I still see some inconsistencies between the different fields:

MetricPkgScore:          0.334186678050244
MetricExportedNamespace: 0.0461191416971477

But with the round trip even if the appearance after apply(read.dcf(file), 1:2, as.numeric) is different (not sure if we need to address this to not confuse users), I see consistent decimals.

We only miss a function to convert back to scores: adding the attributes and class. But we can wait until there are updates on the future of riskmetric package

[aclark02-arcus]

First, thanks @llrs-roche for turning me on to this new-ish workflow. As a test, I used previous scored results for all of CRAN from the [pharmar/riskscore repo](pharmaR/riskscore#9) & converted it to a dcf file. Anyone can use the following code to retrieve it:

# Install dev {riskscore} package from GitHub 
# Note: takes a min to install because there are 'heavy' .rda files stored here
#   which is another reason switching to .dcf files is preferred!
remotes::install_github("pharmaR/riskscore", force = TRUE, ref = "dcf-test")
cran_scored <- riskmetric::pkg_metric_import(
    system.file("extdata/cran_scored_20250812.dcf", package = "riskscore")
)
object.size(cran_scored) / 1000000 # 8 MB

Question for all

Could pkg_metric_export() be adjusted to export pkg_assess() results in addition to pkg_score() results?

Running following code:

 assessed <-
    c("dplyr") |>
    riskmetric::pkg_ref(source = "pkg_cran_remote", repos = c("https://cran.rstudio.com")) |>
    riskmetric::pkg_assess() |>
    riskmetric::pkg_metric_export()

... produces the output below. Notice how the results are trimmed to one element per metric? For example, The assessment values for MetricBugsStatus are usually is a vector of logicals up to length = 30. I don't really care if attributes (like the .recording) get stripped in the process. In fact, that is actually preferred.

DCF Output

MetricReverseDependencies: AATtools
MetricHasVignettes: 45
MetricBugsStatus: FALSE
MetricSizeCodebase: NA
MetricNewsCurrent: TRUE
MetricHasMaintainer: Hadley Wickham <hadley at posit.co>
MetricHasWebsite: https://dplyr.tidyverse.org
MetricExportHelp: NA
MetricRemoteChecks: 3
MetricHasNews: 1
MetricHasSourceControl: https://github.com/tidyverse/dplyr
MetricCovrCoverage: NA
MetricHasBugReportsUrl: 1
MetricDependencies: c("cli (>= 3.4.0)", "generics", "glue (>= 1.3.2)", "lifecycle
         (>=\n1.0.3)", "magrittr (>= 1.5)", "methods", "pillar (>= 1.9.0)", "R6",
         "rlang (>= 1.1.0)", "tibble (>= 3.2.0)", "tidyselect (>= 1.2.0)",
         "utils", "vctrs (>= 0.6.4)")
MetricLicense: MIT + file LICENSE
MetricDownloads1yr: 18597062
MetricRCmdCheck: NA
MetricHasExamples: no applicable method for 'pkg_ref_cache.examples' applied to an
         object of class "c('pkg_cran_remote', 'pkg_remote', 'pkg_ref',
         'environment')"
MetricExportedNamespace: NA