Wish list / roadmap

[larnsce]

This wish list was copied from the README and added here as a to do list:

• Complete codebook of all jmp datasets and of the package (Finalization of the codebook and documentation #2)
• Add label attributes and possibly value attributes to all datasets using [haven::labelled] (Add labels to columns and values #8)(https://haven.tidyverse.org/reference/labelled.html)
• Write how-to documentation and several case studies as article vignettes in pkdgown website How to publish documentation to the web #4 to demonstrate use
• Add WASH in Schools and WASH in Health Care Facilities country files.
• Add use cases on combining with other data sets (national monitoring data, country TrackFin studies, etc.) in pkdown website How to publish documentation to the web #4
• Add tests for data extraction and validation to cross validate country files against world files and different sheets against one another (as an extraction test and internal validation of the data sets). (Validating Excel data extraction #1 and Add standard tests for between PRs and local testing #9)
• Add helper functions to transform world and regional data between the original wide format and a long format.
• Standardize the (long) data format used by datasets in the package.
• Automate rebuilds using file hashes and sampling and a periodic poll of the JMP website
• Prepare package for CRAN submission (https://r-pkgs.org/release.html)
• Prepare an article on “Enhancing the use and quality of official statistics using open source” for JOSS once CRAN publication is finalized
• Python wrapper library for easy inclusion in Python projects

[larnsce]

@nickdickinson

I have reviewed the to do list and other open issues. Some notes and questions:

Complete labeling of all of the datasets

Do you mean something like this? https://strengejacke.github.io/sjlabelled/articles/labelleddata.html

Complete how-to documentation and several case studies to demonstrate use

I would add a very simple README for the pkgdown landing site that's also the README for the GitHub Repo and then more extensive "Articles" vignette pages to the pkgdown site (#4)

Add use cases on combining with other data sets (national monitoring data, country TrackFin studies, etc.)

Combine this with "Articles" page on website in issue #4 and "how-to documentation" point

Add tests for data extraction and validation to cross validate country files against world files and different sheets against one another (as an extraction test and internal validation of the data sets)

That's issue #1, right?

Standardize the (long) data format used by datasets in the package.

Would you say the data is in a "long" format? I would actually argue that this is a wide format and one could standardize one single table as a long-format in which all data is included from all individual dataframes. That long format could then serve as a type of API.

Post article on “Enhancing the use and quality of official statistics using open source”

How about we make it a goal to get this on CRAN and then alongside write an article for: https://joss.theoj.org/

[nickdickinson]

Labelling: yes adding the label attributes. Possibly value attributes. I would avoid an additional dependency unless strictly required though. Also if we do add one, lets ideally stick to tidyverse/haven: https://haven.tidyverse.org/reference/labelled.html
How-to and case studies: basically vignettes. I have used this in a few different consultancies to support D30 and SWA to produce data products among others. So I'd like to give some examples. Indeed having pkgdown site would be best.
Long data form: most are wide, a couple are long (inequalities files I believe). I would like to add a few functions (API if you will) to enable to easily switch back and forth.

As a side note, having a separate package to quickly convert the whole dataset into a star-schema database that can be uploaded into a backend database (SQLite or mssql) would be best. That is a separate but related issue that I should add to the roadmap. This is critical for applications like Excel and Power BI that cannot easily work with the size of the long dataset without massive files. Ultimately, a lot of end-users are in this category. For R users it could also be a more efficient way of working with the data in dashboards like Shiny. I'm not a huge Shiny fan because of the way it is not usually built to scale.

CRAN: Yes, this would be great. I think we are mostly there but there is some documentation of the datasets
Article: Sounds great. Will need to free up some time ;-)

[larnsce]

Thanks, @nickdickinson.

I have reworked the wish list of this issue a bit and had tried to convert some of the to do's to actual issues, but GitHub wouldn't let me.

It might be because I am not a collaborator on this project. Could you add add me?

And then: How would you like me to contribute?

• clone -> dev branch -> feature branch -> PR to dev branch?
• fork to openwashdata/jmpwashdata -> dev branch -> PR to dev branch on WASHNote/jmpwashdata?
• clone -> push to main? ;)

[nickdickinson]

The first would be great. For such a small project we may not even need the dev branch but if we think more people will join up then I think let's set up a dev branch.

Also would be good to setup a few tests to make sure we don't introduce break changes as we move along. I'll create an issue

[larnsce]

@nickdickinson: Can you add Colin as an contributor with push rights to this repo, please?

[larnsce]

Hey @nickdickinson. Just looking at this as I am still in the process of completing the report for the washcollab project. Are you still intending to continue developing this package?

[nickdickinson]

@larnsce definitely open to it. Do you/GHE want to take it over instead? I think either is good. I don't want to be the bottleneck! I have some limited time but probably not much in the next month. I can also advise.

I've been following up with WHO to try to get the stata files. I will definitely continue to chase that. The web API to me is not reliable enough since it is just a coupling between the website and it's visualizations. I would want that to be documented in API documentation with versioning to build on it.

Also available to chat on Matrix