A lot of extract-transform-load (ETL) work requires unloading and un-transforming first. Rather than \ETL\, it’s \L_A^{-1}T_A^{-1}ET_BL_B\. What the data provider did is \A\. What you want to do is \B\. The data provider gave you a “dump” of their data. You don’t know what it means. If you did, you could extract (subset) from it according to your needs – filter entities by some meaningful criteria and collect selected attributes.

Wikidata uses opaque identifiers for its catalogued information resources. For example, the statement “wd:Q42 wdt:P69 wd:Q691283” may map to the label sequence “‘Douglas Adams’ ‘educated at’ ‘St John’s College’” with a language-locale preference of English-US. Opaque naming is wise for internationalization.

I recently encountered the Archival Resource Key (ARK) identifier scheme and read its latest draft specification. ¹ Its thesis is that the persistence – i.e., long-term identification and access – of any information resource – i.e., an “object” – is a matter of service.

Imagine a data system modeled as three parts: an interface, a processor, and a repository. The repository “contains” information. The processor receives symbol streams to alter or retrieve information from the repository, and the processor outputs symbol streams. The interface is the medium, the opaque surface, of symbol-stream exchange between you and the processor. ¹ What information is “in” the repository?

After my last note on identifiers, Leo Talirz pointed me to a great riff ¹ on Tim Berners-Lee’s classic note ² on “cool URIs”. In the “Cool DOIs” article, Fenner breaks down a DOI into three parts: proxy, prefix, and suffix. A proxy is a server that maintains a map from prefixes to registrants. Example proxies are https://doi.org/ and https://hdl.handle.net/. An example prefix is 10.5281.

Data protocols vary over project lifetimes, and many projects involve parameter sweeps. You might see filesystem directory structures evolve naming schemes like the following ¹ : # let's not overthink this at first. concentration_A_0.25/ # hierarchy is good, right? concentration_A/0.25/ # paths are getting too long. conc_A/0.25/ # change to percentages. clever!

Approaches to data citation may span classes of big-O complexity, for both space (memory/storage) and time (compute/transfer). Dataset revisions may be minted and persisted without any delta encoding / structural sharing. The main mechanism of reproduction for citations in this case is restoration. Space complexity is high, as storage needs are high.

The hourglass model is an approach to layered system architecture where a middle layer is intentionally constricted in order to support flexibility in the implementation of layers above and below.

Against what bases are queries against your data evaluated? If you only expose a single “data base” that changes over time, then data citations cannot be a combination of query and basis. When citing a passage in a book, the edition/variation of its publication, i.e. the thing that is assigned an ISBN, is the basis. Optionally, a citation may include a “query” against this basis – a page number, page range, chapter number/title, etc.

Many tests use oracles , where you know the answers for some inputs and you check those correspondences. To cover more of the input state space, you can generate random inputs and check some properties for each corresponding output. You don’t have an enumeration of exact answers like with oracles, but you can check things like the output always being greater than zero, etc.

In a collaboration, data objects are produced at many sites. To make the data objects findable, you may steward a central, searchable index for their metadata. How then do you make the data objects accessible for download? One common solution is to centralize the custodianship – have all sites upload copies of their data objects to a central store. The central store may partition storage across several physical servers behind the scenes (e.g.