Penny muses on inter-reliance of researchers working in different stages of the research workflow. 

Last week our paper on capture of FAIR-ready data on the FAIMS mobile platform was released in the Journal of Computer Applications in Archaeology (Ross et al. 2022). In it we step through each of the FAIR data principles: 

• Findable with unique and persistent identifiers and rich, searchable metadata

• Accessible via open, free and/or standardised protocols

• Interoperable with machine-readable metadata

• Reusable having been shared with all relevant contextual data, ready for reuse by other researchers

While these are well known concepts in eResearch communities, they have had little traction to date in archaeological research. That said, they are compatible with a long-standing willingness to share archaeological data. 

As we note in the paper, there is an inherent tension in the need to convince the creators of data to share it in a way best-suited for consumers of those data. When data are shared directly and in small groups, this persuasion is a straightforward task. Creators can discuss user’s needs at the time of distribution. When researchers create data for ‘future users’, they must anticipate the needs of unknown agents, asking yet-to-determined questions using unknown tools. It is for this reason that agreed protocols are so important (and it’s handy if one of those agents is a machine). But, this effort can create a burden on the data-creators who are almost certainly subject to the pressures of data collection efforts. 

Of course, few researchers embark on careers in knowledge-making without some modicum of good will to share their data with the world. Not all realise that simply sharing the data is not enough. 

Our efforts with the FAIMS Project—which commenced before the FAIR Principles were established, but in large part was to achieve the same goals—has been to make the process of capturing these necessary metadata as painless as possible. As Shawn wrote in April:

It is much easier to produce FAIR data if FAIRness is ‘built in’ from the beginning, rather than ‘bolted on’ at the end. (FAIMS blog post)

While, as data collection software the FAIMS Mobile Platform does not play a role in the accessibility of data, we have:

• supported the allocation of unique, persistent identifiers such as the IGSN and capture of key project- and dataset-level metadata (F=Findable);

• provided the means to connect connect local controlled vocabularies to published vocabularies, thesauri and ontologies (I=Interoperable);

• supported the creation of rich and varied record- and value-level metadata to make data easier to use (R=Reusable)

Despite this, we found that: 

Features that helped researchers accomplish their own goals around timely analysis and publication of traditional outputs were widely used, while features that focused on making data FAIRer to benefit others were often neglected. While improvements to field data capture software and more collaboration between archaeologists and data specialists can make FAIRification easier, the major barriers are sociotechnical. Until a disciplinary culture emerges in archaeology where production and publication of FAIR data are normal and rewarded, these activities are unlikely to happen at the necessary scale, limiting the amount of genuinely reusable data available to address our discipline’s grand challenges. (Ross et al. 2022) 

The issue of rewarding researchers for performing thankless tasks such as FAIRification was discussed at the recent eResearch Australasia Conference at the Brisbane Exhibition Centre last month. It was raised in a number of sessions on the publication of vocabularies, finite citation of enhanced and amended datasets (Klump et al. 2021b) and the use of adoption of ORCID IDs. I was chuffed to see that our institution, Macquarie University, is an outlier in its stellar uptake of ORCIDs for staff members (Porter 2022). This came about through positive change management—as (Nicholas et al. 2009) argued when it comes to persistent identifiers and other critical components of FAIR data: the policies not technology alone are important. 

It is to this end that the archaeological members of the FAIMS leadership (Shawn, Adela and and I) are working with Jens (as a founder of IGSN) and Rorie Edmunds of DataCite to build a community of practice to support the adoption of IGSNs in archaeological practice (Klump et al. 2021a). Here, we hope to lean on expertise of researchers from other fields who have successfully implemented this kind of persistent identification in their research domains to enhance the FAIRification in archaeology. In turn, we hope that some lessons learned by our growing community will be shared with those that come.

Share

References

Klump, J., K. Lehnert, D. Ulbricht, A. Devaraju, K. Elger, D. Fleischer, S. Ramdeen & L. Wyborn. 2021a. Towards Globally Unique Identification of Physical Samples: Governance and Technical Implementation of the IGSN Global Sample Number. Data Science Journal 20: 1–16. https://doi.org/10.5334/dsj-2021-033.

Klump, J., L. Wyborn, M. Wu, J. Martin, R.R. Downs & A. Asmi. 2021b. Versioning Data Is About More than Revisions: A Conceptual Framework and Proposed Principles. Data Science Journal 20. Ubiquity Press: 12. https://doi.org/10.5334/dsj-2021-012.

Nicholas, N., N. Ward & K. Blinco. 2009. A policy checklist for enabling persistence of identifiers. D-Lib Magazine 15: 1082–9873. https://doi.org/10.1045/january2009-nicholas

Porter, S.J. 2022. Measuring Research Information Citizenship Across ORCID Practice. Frontiers in Research Metrics and Analytics 7: 779097. https://doi.org/10.3389/frma.2022.779097.

Ross, S., B. Ballsun-Stanton, S. Cassidy, P. Crook, J. Klump & A. Sobotkova. 2022. FAIRer Data through Digital Recording: The FAIMS Mobile Experience. Journal of Computer Applications in Archaeology 5: 271. https://doi.org/10.5334/jcaa.96.

Subscribe now

A FAIMS 3 Poster (and general project update) at eResearch Australasia

Penny and Jens are busy this week as eResearch Australasia in Brisbane, presenting our poster (https://osf.io/5karu/). I am told, that it was ‘Highly Commended’ for condensing a lot of information onto a single poster and providing all the metadata.

Did you see our poster at eResearch Australasia 2022? Tell us what you thought!

Leave a comment

Dev Update

The major news as our development period draws to a close is sixfold:

1. We have started to commit parts of our UI Refresh

2. We are working on better visualisation and user experience around relationships

3. We have demonstrated branching logic

4. We now support Google OAuth2 logins.

5. Regression testing on our iOS build has found no novel bugs.

6. We have basic QR code parsing working!

We have about 1 week left1 until feature freeze and so we are gliding2 to the pragmatic end of our development time for this stage of the project. Still, have some screenshots:

Progress!

We will be talking about our 2023 plans in another blog post.

Subscribe now

Articulating thoughts about server architectures

My thoughts around our architecture design, inspired by A Brief History of Kubernetes, Its Use Cases, and Its Problems and a conversation we had with an amazingly knowledgeable colleague at CSIRO.

I have spent my “free time” in the last week trying to move our present set of Docker containers into a better and more sustainable architecture. We have three major components:

1. The FAIMS Application. This is either a static site or a natively compiled app.

2. CouchDB. It’s just CouchDB with minor configuration.

3. FAIMS3-Conductor — a lightweight authentication and role-management server. It provides the JWT token to FAIMS to be able to talk directly to Couch.

At present count, I have (about) 7 instances of the above set running on two machines. Each component has a Dockerfile and configuration of the components is handled through docker compose files to define how the three containers relate to each other, docker compose files using `extends` to implement a server-specific design, and environment files holding necessary variables per instance. The major complication with deploying FAIMS is key-generation and sharing between components, necessitating a bash script to orchestrate the orchestration layer.

Right now, the instances are on two VMs running on a rented server on campus. Bash scripts and ssh provide a pragmatic and comfortable reality. This reality is mirrored by our app deployment “strategy” wherein I have a bunch of shell scripts calling Fastlane when I need to deploy a specific Android or iOS build.

All of the above makes me a bottleneck with a bus factor of 1. This is a bad number. Given that Penny and I will be on leave throughout December and January — we will have an experimental validation of that factor arriving soon.3 As such, as the weather warms, thoughts turn to leave, and we exhaust our current development budget, system architecture and design becomes a more pressing concern.

Tilting at Kubernetes

Thus, the obvious4 next step would be to use Kubernetes to fully automate deployment of all systems and servers in a scalable fashion to be cloud-provider independent and reduce system administration load.

Yes, I can hear you laughing from here. I was younger and more naive two weeks ago. You can stop laughing now. I should have listened when a close friend told me how much more than me a Kubernetes engineer makes.5

Here is what I’ve figured out so far. My confidence level in what I'm saying is: "Roll for a sanity check." Kubernetes6 is a platform for declaring how ephemeral containers (Pods) should behave with regards to each other in a Service with specific deployment options committed as their own file. It is infrastructure as code. Changes to running machines are achieved through changing the deployment yaml file.

I must admit that I’ve not spent so much time on an endeavour with so little success for some decades now. Trying (and ultimately failing) to get a minimum viable example deploy over the moments of free time and no-focus over the last two weeks reminded me of some of the more trying times as an undergrad.

In conversation with a colleague at CSIRO yesterday, he revealed that Kubernetes can be powerful, but we should:

1. Expect to employ a full time engineer to manage any Kubernetes infrastructure we have; and

2. Only use Kubernetes when our system administration load becomes too complex for our current tools.

Here are some choice quotes:

Plan to use "Kubernetes for a scalable base load with everything running, with burst workloads" and "If your base load is closer to 0, Kubernetes is harder to justify the cost of."

Effectively, it’s an amazing tool for scaling out complex infrastructure. The folks at https://bioinformatics.csiro.au/variantspark/ achieved amazing efficiencies at scale with Kube. But it was an always-on complex workflow that already had significant amounts of system administration and load being thrown at it. Trying to go from “a few shell scripts” and a load of “4 projects have used us” to “a successful Kubernetes” deploy was unwise.7

"We'll need to hire someone to maintain a Kubernetes cluster" and "There are plenty of people who have gone backwards from Kubernetes"

Our very kind colleague provided useful veteran knowledge and expertise as to the minimum necessary buy-in needed to do anything useful with Kubernetes. While very powerful, it’s a powerful tool best employed when needed, rather than at the start of things.

I’m sharing these thoughts here so that other academic DevOps people wearing many hats may learn from my mistakes.

Current thoughts

I’m procrastinating from actually building out some test deployments on The Cloud: Conductor as a serverless appliance, our app as a true static site on Github Pages, and a CouchDB container where I can’t simply reach out and touch it from the shell.

Once I build those demonstration layers and prove that the conductor startup time is sufficient for us to be able to offer it as a serverless container, it was suggested that I look at tools like Terraform to progress down the infrastructure-as-code route without trying to go from 0 to 60 in negative time and non-euclidean space.

Any thoughts as to other useful tools that I really should have already had known about?

Leave a comment

What I’m reading

• Many tributes to the late Bruno Latour:

  Justin E. H. Smith's Hinternet

  A Locus of Care

  Down in the crypt of the basilica of Saint-Maximin-La-Sainte-Baume, in the South of France, there is an exquisitely rare object. It is a skull, behind a wall of glass, and it is described by two separate and very different labels. The one label tells you it comes from a woman in her fifties, likely born in the eastern Mediterranean in the early first ce…

  Read more

  3 years ago · 26 likes · 6 comments · Justin E. H. Smith

• Gelman’s Museum of Scholarly Misconduct

• Is the AI spell-casting metaphor harmful or helpful?

• Incentives: The Most Powerful Force In the World

• Twitter Polls: Evidence is Evidence

• Writing with Copilot

• The Case For, and the Case Against, “The Case Against Education”

• Short story: Lost and Found by M. L. Clark

• Short story: Holiday Horrors: Feast or Famine Rulebook

• Short story: Ten Steps for Effective Mold Removal

• A New Topography of Philosophy: Analytic, Continental, and Philosophy of Science

• Publication Laundering

• The Hierarchy is Bullshit, We Can Do Better

On 17 to 19 September a team of archaeologists and MQ students returned to the Jamison Valley, Blue Mountains, to conclude a survey of shale-oil mining ruins as part of the ARC Linkage Project: ‘History, heritage and environmental change in a deindustrialised landscape’.

Led by Tanya Evans, the project is weaving together multiple strands of archaeological, archival and oral history research to explore daily life for miners living in the valley in the late 19th and early 20th centuries. It is more than a data-collection exercise, we have concrete goals for data integration, distribution to community interest groups and management planning with the Blue Mountains World Heritage Institute (BMWHI) and the NSW National Parks and Wildlife Service.

It was the tenth deployment of a FAIMS digital recording platform in the Jamison Valley since June 2014 when we first began working with the BMWHI to record the remains of the Bleichert Ropeway.1

While we recorded on FAIMS 2.6 module,2 we took a demonstration FAIMS 3 notebook for a test drive.

This was the first time back in the field since April 2021 (two planned seasons of survey were cancelled owing to COVID restrictions and, more recently, floods) and the author—best known as an artefact specialist (= not a field archaeologist)—confesses to being a little rusty on the field preparation. I had the tablets, the cords and a new gear bag but updating user lists on the FAIMS 2.6 server, pushing to tablets, flushing test records, forcing updates, checking version numbers were all small hurdles forgotten in the modern architecture of FAIMS 3. In our rebuild these administrative tasks can be done on device and design tweaks are automatically refreshed (while online).

It was nice to be in the field, of course, and to try out some new hardware: the Bad Elf and our new Samsung A7 tablets. As far as performance goes, the A7s were superior to our aging, heavy, screen cracked tablets used in previous seasons. They were faster and the battery life was good—no nail-biting scramble to record one last thing before the tablet on 1% power dies in your fingertips—but we did struggle with the screen visibility. In low light, under dense canopy, the screen was only visible if held at particular angles. This was fine once you found the ‘sweet spot’ but did make it challenging when glancing over the shoulder of trainees learning to use the system.

As we wrapped up the final season of survey, our attention is turning to the next stages of the project: excavation, management planning and outreach. Here we face some well known challenges: aggregating data from all seasons (already done); auditing and refining data from all seasons (in progress); and integrating survey data with an amazing trove of archival data collected by historians Anne Cootes, Tanya Evans and Lucy Taksa. It is this last component that will inform the next stages of management planning and more importantly underwrite our efforts to share our research findings in a meaningful way via a public-facing app. It will almost certainly have a core geospatial component but needs to be multi-media and capture relationships—and specifically—the certainty of geolocations and relationships so that when we transparently communicate that certain events may have occurred in the locations we have uncovered.

To this end, we are exploring ways to import the refined and consolidated data from FAIMS 2.6 into FAIMS 3.0, and extend that notebook—currently designed for physical survey—to include some basic archival and biographical and historic event sequencing.

This is not what we ever considered we might do with FAIMS 3.0. It will require some tweaking to the roundtrip feature set we currently have but it would be a very useful addition to the software suite. As an offline data-collection tool, we are focussed on supporting efficient data creation and consider it to be the start of the journey, not the destination. But as the software evolves, and the cross-platform nature of the system becomes second nature we are seeing more stages in the research workflow that needs flexible, extensible and trackable data capture efforts.

Subscribe now

References

Parkes, R., S. Ross, A. Sobotkova, T. Evans, P. Crook, S. Lupack, G. Karskens, F. Leslie & J. Merson. 2018. Ruined Castle shale mining settlement, Katoomba NSW: report on a pilot survey. Australasian Historical Archaeology, 86–98. https://trove.nla.gov.au/version/263328093.

Crook, P., S. Ross, G. Burnett, P. Shadie & Phil Hammon. 2015. Bleichert Ropeway, Katoomba: 2014 Survey Report. Katoomba, N.S.W.: Blue Mountains World Heritage Institute. https://www.bmwhi.org/bleichert-ropeway.

Ross, S., A. Sobotkova, P. Hermankova, C. Nassif-Haynes, P. Crook & B. Ballsun-Stanton. 2018. Field Survey of the Blue Mountains World Heritage Area, NSW. https://github.com/FAIMS/bmwhi-survey-2018.

Dev update

But first, a dev update! Besides quite a lot of devops work on CouchDB and our conductor integration (delayed by the events of the last few weeks), we’ve been:

• 🎉 We just got authentication working on the compiled iPad app! (Still quite a lot of testing to do, but this was a problem that has plagued us for months.)

• Whacking through bugs;

• Testing branching logic (the ability to control field and view visibility conditional on prior values);

• Working on improvements to form-relationship navigation (to make both parent-child and linking relationships more usable); and

• Doing quite a lot of work on the UI.

We’ve also redoubled efforts for iOS integration. We have … 6… weeks left until feature freeze1.

Data Collection Workflow Discussion

As a preface, these diagrams are Graphviz specified diagrams produced by the Sketchviz online service and saved as github gists. We live in a strange world. (The ‘sketchy’ line-rendering is a function of a svg transform of the sketchviz app-presentation.)

Fieldmark starts with workflows, preferably established ones. One thing we’ve noticed is that more data and more useful data is produced from a project that knows what it wants. A theoretical form, or one copied from a standard template, is seldom used in its entirety, and often omits important (meta)data, which accumulates as implicit knowledge. The complexity of implementing unused fields is wasted, while ‘missing’ fields degrade data.

The field director (or delegated person) makes the notebook specification. One branch, planning, makes the workflow to suit the data needed. We don’t support multi-person notebook editing at present, so one and only one person should be responsible for the notebook design. This limitation will change in the future.

We then deploy the notebook on appropriate servers. This process is, at present, entirely manual and mediated by Brian. We look forward to automating this in the next 6 weeks.

At the same time, the team should prepare to record data. The team can deploy Fieldmark onto several devices through various channels: Android, web, and iOS2. As they are deploying devices, the team should also be registering and communicating their usernames to (again) Brian. This process of adding users to a notebook is also something we plan to automate really soon now.

[Everyone signs in, and does' pre-field prep, not pictured]. I very carefully left the boring 'pre-field prep, pre-field prep-prep, and meetings about meetings to prepare to prep for fieldwork'3 out of the above diagram. But yes, some time between notebook instantiation and going to the field, everyone needs to sign in and make sure their tablets have the notebook. Pragmatically, this can be done before leaving for the field, in the field house, or on a wireless 4G dongle4.

The offline zone.

Once synchronised, data collection can begin. Users can bring tablets and phones and laptops5 out into the field to collect data. Specifically, we support the full CRUD: record creation, retrieval, updating, and deletion. We’ve demonstrated using external Bluetooth dongles and support onboard camera + external file attachments for photos and multimedia. Thus, we have a team, while offline, collecting geospatial and multimedia data in addition to structured data as specified in their workflow.

Useful aside on the nature of offline. One of our fundamental features of FAIMS 2, and one of the things we’re working towards in Fieldmark,6 is the ability to use a local network for syncing and other 'online' actions. We hope that with a Raspberry Pi, some solar cells and dongles, researchers will soon7 be able to bring a local area network with them, past basecamp and onto a dig or survey or sampling trip. Just today I set up replication between two of our CouchDB servers to aid in tracking down a performance bug, so all signs point to this offline-online capability being possible and useful. It would be a step up from a NUC in a pelican case mounted in a truck.

Returning to … the online zone

It is likely that for the next year or so, exports will be accomplished through custom scripts, rather than through functionality built into the app itself. The rationale for this is:

1. Tradition! Specifically, in FAIMS 2, we were able to accommodate many data formats by having the server run scripts8 to extract data, tweak and transform it in any one of many ways, then export a compressed file with many data files, photos, and folders within. This data-flexibility is one of our main selling points, but to accomplish it--we (I) will need to tweak data exports more frequently than app updates allow.

2. Complexity. Building this super-flexible and extensible capability into Fieldmark while preserving security is something of an expensive ask. Instead, by isolating this complexity to the outside, we can run arbitrary data transformations on something like Cloudstor’s SWAN instance while preserving the security of the original data.

3. Money! We have so much to do that… as I was able to write an exporter… I did. When this feature is funded, we’ll bundle it with the app.

Regardless, we don’t expect most folks to have a working Python interpreter on their system, and so this will be an online-mediated process as they connect to MyBinder or Cloudstor, as the instructions we will soon9 write will instruct.

Leave a comment

Technology Graph Discussion

This 'semi-directed'10 graph shows how:

1. various technologies relate to our online/offline stuff;

2. gives a rough11 network topology; and

3. shows most (ish) of the high-level dataflows12 as communications between technological components.

Pragmatically, we have 5 major components:

The Fieldmark web-app: the app, itself, is written in Javascript (NodeJS ← React ← Formik ← Yum, Material UI, OpenLayers, Capacitor, Moustache). When presented on the browser, the browser receives a static site (HTML + compiled/minified JS + CSS + binary assets). The app uses PouchDB as a local, offline-capable, NoSQL datastore. It reaches out to our server (we call it the Conductor) to generate a JWT authentication token.

Our lightweight server, the Conductor, handles role management and authentication. It however, does not perform data exchange. The Conductor mints tokens, will eventually be able to moderate, edit, and create notebook designs, and will handle being a wrapper for our CouchDB datastore for other clients and non-Fieldmark users.

PouchDB uses the JWT token to replicate with a central Apache CouchDB datastore running in a Docker container13. The reason we chose CouchDB/PouchDB was because full replication (copy+sync) running in an 'eventual synchronisation' state (i.e. when the database can see the network again) was very exciting to us. We didn't have to write an offline-capable data synchronisation protocol.

Once all notebook metadata and data are synchronised, a properly authenticated user can load visible14 notebooks and visible records. Once this initial sync is performed15 the app is entirely equipped for offline fieldwork.

The Fieldmark mobile app is all of the above, except instead of NodeJS, Ionic’s CapacitorJS transpiles the javascript + extra bits into native code for iOS and Android. Pragmatically, it’s the same technology used by Electron to wrap a web browser and an extremely specific set of pages in an executable wrapper. As a hybrid native app, however, we can access the device-specific APIs for geolocation, camera, and Bluetooth — functionality critical for arbitrary workflows. Capacitor offers many plugins in this vein.

Roundtrip and Export scripts16 are presently written in Python and connect directly to the CouchDB servers. By using our own library, we can translate the form -> record -> revision -> attribute-value-pair NoSQL appropriate format of our internal datastore into dataframes appropriate for manipulation and export. These scripts can create new revisions (changing field values and record metadata like deleted flags) as well as exporting single entries or an entire workflow's datastore. Once in a more computable format, my scripts then rename photos and files, organise everything, and export to csv, xlsx, KML, and GeoJSON.

Leave a comment

Stuff I’m reading

• Quill & Still: Sophie Nadash And The Quite Nice, Very Cozy Divine Transmigration is an utterly lovely academic power fantasy that reminds me of Graydon Saunders’ Commonweal series. If you like web serials and you’re an academic, I suspect you’ll identify quite strongly with the protagonist.

• Xe on Push notification two-factor auth considered harmful.

• Patrick McKenzie on bank branches.

• A short story called ‘Recipe’ that also features an academic somewhat down on their luck.

• A travelogue of the Nevada National Security Site (Parts 1-3 currently posted).

• An interesting look at the techniques of ‘InnerSource’ (software development as applied to enterprise protocol and procedures and how to engage external contributors.

• The White Tree of Gondor: A brief overview of Modern Ukrainian SF&F.

Subscribe now

Status Update

Our devs are hard at work, updating our server code for role management, making persistent fields (so that field values entered in one form instance persist through the next record), and having branching logic/conditional visibility ready for QA in the coming week.

We are also dealing with the natural consequences of multiple teams taking FAIMS3 to the field for the first time, which… goes some way towards explaining the shortness and delay of this post.

We now have four total notebooks which have been used to collect genuine (not test) records!

Where did August go?

If anyone finds a missing August, and if they could return it to me, that’d be great.

Stuff I’m reading

• Simon Willison on Stable Diffusion, the “text-to-image diffusion model”

• Morgan Housel on Rare Skills

• Roman Army? or Roman Armies?

• C. Thi Nguyen on Teacher, Bureaucrat, Cop

• Sixty Symbols on The Panic Paper

• John Danaher on Woozles, Academic Publications, and a lack of Rigour

Status update

Planning continues for our November production release. In the last two weeks, we’ve mostly been cleaning up and releasing to production for PSMIP Data. We’ve also started work on redoing our authentication services to make sure we can hook into multiple OAuth providers.

You can see our latest work on Google Play, so long as you’ve registered with DataCentral and signed up on our demo registration form so we can give you access!

In something that makes me personally quite pleased, one of our collaborators at Macquarie University (Tess Nelson) independently developed a second Groundwater Pump Testing notebook on her own initiative, without needing support from me! We have already met FAIMS2’s record for independently created field notebooks!1

FAIMS 3 - Electronic Field Notebooks

In the field with the last FAIMS 2.6 module

In this blog post, we handover to Macquarie University researcher Tess Nelson to reflect on the last FAIMS 2.6 module we will ever design. Working with supervisor Professor Grant Hose, Tess and their team have taken this module out multiple times over the last year to do groundwater sampling and vegetation observations…

Read more

4 years ago · 2 likes · Tess Nelson

Demo Day

The Electronic Field Notebooks team also attended the MQ Incubator demo day last Thursday! It was mostly a chance to practice our short pitches and realise that our traditional hour-long demo presentation … doesn’t map to running a corner of a table and trying to get folks wandering around interested in our product. Many lessons still to be learned there.

For online demos, we’re thinking of making 5-minute videos around various features of FAIMS 3. What do you folks think about that length. Too long? Too short? What would you want to hear in a pitch for electronic field notebooks?

Leave a comment

A response to ‘HARKing to Big Data’

First, to “HARK” is to “Hypothesise after the results are known” (As discussed in our book chapter discussed in Dev diary 6):

FAIMS 3 - Electronic Field Notebooks

Dev Diary 6: A philosophy of Digital Archaeological Tools

Brian gives a dev update for mid April, and descends further into the philosophy of technology and archaeology. (Madness intensifies…) Dev update We have just begun a new tranche of development work for CSIRO, focusing on notebook authentication, row visibility controls, new work on the exporter, and atta…

Read more

4 years ago · Brian Ballsun-Stanton

Here, I have a few thoughts in response to Jeremy Huggett’s HARKing to Big Data:

None of these [hypothesis generating practices] sound like good practice, especially the idea of deliberately (and quietly) setting hypotheses aside that don’t work out. But it isn’t quite so simple. Forms of HARKing are quite common in archaeological practice. We’re familiar with searching for patterns within our data, deciding which ones seem to be useful, and then hypothesising about what they might mean about past lives and activities. It is, after all, related to the process of scientific induction, where conclusions are drawn from patterns observed in the data, although this is not without its problems (see Smith 2015, 19, for example, and the discussion in Ross and Ballsun-Stanton 2022).

In our discussion in Digital Heritage and Archaeology in Practice, we say2 (page 3):

Indeed, “using current results to construct post hoc hypotheses that are then reported as if they were a priori hypotheses”, “failing to report a priori hypotheses that are unsupported by the current results” (Rubin 2017), or “presenting exploratory work as though it was confirmatory hypothesis testing” (Fraser et al. 2018), is considered “hypothesizing after the results are known” or HARKing (Kerr 1998). HARKing is always a questionable research practice if it is unreported, although disagreement exists about the acceptability (or even desirability) of careful and transparent post hoc analysis in deductive research (Rubin 2017; Hollenbeck and Wright 2017). In a 2018 paper surveying over eight hundred ecologists and evolutionary biologists, 51% admitted to HARKing (Fraser et al. 2018). Fraser also notes in passing that when published papers fail to disclose a priori hypotheses (or if there were a priori hypotheses), it becomes difficult to judge whether HARKing – or the conflation of postdiction and prediction more broadly – has even taken place. As such, the a priori articulation of hypotheses required by preregistration (or the explicit statement that research is inductive) can help to combat this species of questionable research practice.

A brief recap of research modes:

• Hypothesis-testing research, deductive research, is taking a known general case and deriving conclusions from it. Deductive reasoning is tautological. Deductive, predictive approaches in research are useful — as testing one’s assumptions and if the general case holds in a particular instance is one way of falsifying a risky statement3.

• Hypothesis-generating research, inductive research, is reasoning from specific instances to a general covering case. It is the practice of keen observation and observing trends in nature. Case studies are a great example of this sort of research in a qualitative context.

• Abductive research engages in rapid hysteresis between inductive and deductive research, exploring and narrowing down a possibility space for future investigation. Pragmatically, I expect some research projects utilise abductive methods either during initial conception or during writeup when scrambling for a literature. I have seldom seen the practice used explicitly and with prior intent.

Huggett explores around the concept of Corrigibility when he says:

So there’s no problem with big data mining and allied artificial intelligence related methods of analysis which hypothesise after the results, as long as we are transparent about this, then? Not entirely. If the algorithms used are black-boxed, or the internal procedures so complex as to defy understanding, and the systems incapable of explaining their reasoning (e.g., Huggett 2021, 424-428) then the transparency required of ‘good’ HARKing cannot be achieved. There is, therefore, no substitute for human intervention in the process of analysis, to understand and to evaluate the outcomes, and to determine whether the patterns and relationships identified are actually valid as well as useful. (Emphasis mine)

A corrigable “AI” (or ML system) is one that is inspectable and interrogatable as to its methods. Certainly, the current practice in ML training is to reserve a portion of the training dataset for verification to ensure that the system is being trained appropriately. Moving away from AI, to “Big Data4” this sort of correlational investigation fits poorly into the inductive mould. Documenting correlations in “large” datasets may be useful, if not real. Huggett then discusses the need for ground truthing:

Of course, human intervention is a feature of many current archaeological applications of neural networks, data mining etc., typically evidenced in a concern for ground truthing and correcting the algorithms and the patterns that the systems latch onto. But this concern is often seen in terms of training the system, implying that the responsibility for the outcomes will ultimately be transferred to the system and in the process setting aside the role of the human expert in validating the patterns and the conclusions that may be drawn from them.

In one sense, we can consider this an abductive process: the inferences produced by machine are validated “in the real”. Having just supported a team going on a survey, I see some similarities with the normal field experience of folks with FAIMS modules/notebooks: there tend to always be tweaks required when the notebook hits the field that were not previously anticipated in the research design.

From my reading of this essay, I see a few themes:

• Using algorithmic black boxes to conflate correlation with causal induction

• Unreported archaeological “tweaking” of protocols while on the dig site as a form of HARKing and the Gattigla discussion of setting aside hypotheses while observing

• The danger of using “big data” claims as an invisible HARKing practice that is proof against corrigibility/transparency while maintaining the pretence of hypothesis testing activities.

First: I believe that sufficiently advanced applied statistics (i.e. ML) can indeed find causal links amongst correlational links. Specifically, that while every correlation is not a causation, every causation has correlated interactions. (By virtue of being a cause, it causes a change in a dependent thing which could be described as a correlated pattern.) However, from my perspective of making field notebooks—Archaeological data is entirely a small data domain. The current means of investigation for survey, excavation, experimental do not admit to the kind of automation at scale that would create an environment for standardisation of data to the level needed for ML. Not to say that other disciplines use ML techniques (Remote sensing comes to mind, as well as vision techniques in pottery sorting and apps like ArchAIDE) are not useful—they certainly aid in novel knowledge creation, but that the present reward systems create few incentives for consistently encoded knowledge representations of archaeological results.

This is entirely OK. Not everything needs to have a Netflix recommendation engine prize5 as the basis for interaction and knowledge production.

As I write this, it feels like an error is being made (either by me, my reading, or Huggett). Looking at the ML techniques above, they are all tools. Right now, they are tools used by humans towards knowledge production, academic goals, and as an excellent excuse to play in the dirt6. Huggett argues:

Gattiglia essentially proposed that theory is set aside temporarily and comes back to the fore following the data analysis, whereas I suggested theory cannot be set aside – like it or not, whether recognised or not, theory is involved at every stage from the recognition, selection, collection and recording of the data onwards[.]

Drawing on my research in the Philosophy of Data (Ballsun-Stanton 2012, 348)7 the structure of the data store informs the shape of the knowledge. Huggett says that theory is involved in every stage—and there I agree. But creating a data store and letting a ground-up investigation of the data provide the impetus for an abductive investigation of what a site or landscape tells you strikes me as a valid approach?

Specifically, we have not yet achieved Artificial General Intelligence8—which means that claims about system-based knowledge production are not valid claims. While triplestores and other knowledge representations may be capable of deductive reasoning, our tools exist to facilitate our searches. Our methodologies may be as meta as we like: from going to a dig with a noticeably clear idea of its history and well-formed data collection methodology to approaching a landscape with a nice notebook in hand for a good session of Slow Archaeology. The tools we use (a nice leatherbound notebook, ML based vision systems, electronic field notebooks, … even excel) are not our results. Both Ackoff and Tuomi’s hierarchies/ontologies of data (see Ballsun-Stanton 2012 for a longer discussion of ontologies of data, information, and knowledge) admit to a relationship between knowledge and data—be it hierarchical or cyclic. An uncritical use of a tool is never a good idea. But making a space for inductive or abductive research as part of the knowledge production cycle remains a theoretically valid knowledge creation activity. Correlation may still have useful pointers for further investigation, even if archaeology has no risk of producing Big Data.

Fundamentally, I agree with Huggett’s conclusion: we must never forget the human researchers are responsible for their outputs. A machine, a tool, an ML approach is only as effective as its designers and reflects them and their theoretical approaches to knowledge—even if the designers make some sort of claim to objectivity.

Subscribe now

Stuff I’m reading

• (Youtube) Joseph’s Machines — Pass the Wine

• Technocracy and the Space Age

• In terms of ML, I got a 6/10 when trying to differentiate Daniel Dennett from a computer

• On ritual, friction, and formality when shipping software

I discovered a while ago that all those errors and bugs that only appear when you demo something to an audience also magically appear when you record yourself demoing it to nobody. Maybe narrating a feature to a pretend audience takes the blinders off enough that you notice little mistakes you wouldn't have otherwise.

Karaterobot via Simon Willison

The final countdown

As we enter our last development cycle, we are closing in on the final feature set that FAIMS will be sporting early next year. Our “expensive sentence” (which has turned into a paragraph1) starts:

To finish the work of developing FAIMS 3 as an acceptable successor to FAIMS 2.6. In order to this, our goal is to achieve a polished production/operational implementation of FAIMS 3 which meets the goals of our original ARDC contract…demonstrating a mature OSS product suitable for task.

Features on our August–November Roadmap

Some (much smaller) subset of this list is what we’ll be working on for August–November, but here are the features which presently have our attention. Comment below if you really want one of the features or have ideas about a feature that will solve a problem you’re presently having!

In (very, very) rough order:

• “First-class” iOS support (see the next section for our present workarounds)

• Offline Server support

• Authentication, notebook-specific sharing controls, and user-provisioned notebook deployment

• Improved interface for child entities

• Offline basemaps and custom symbology for collected data per layer

• Field inheritance and persistence

• Branching logic for fields and views

• External bluetooth GNSS providers (see below)

• QR code scanning and record loading via QR code

• An in-app record history and undelete view (we already have that via external tool)

• Timestamp fields to reflect record creation and/or a button press to record when an experiment was run

• More work on external bluetooth

• Sharing/exporting/importing notebook designs in the notebook designer

• Live, user-initiated data+metadata backups and restores

• Search improvements (ultimately faceted search by data and metadata and paradata2 attribute, but … not now)

• Picture vocabularies as per FAIMS 2

• A more robust data API

• Tabular record view for viewing/editing in bulk

• Having controlled vocabularies limit their contents as a more sophisticated form of branching logic

• Federated logins

• Sync status button3

• More polish on the notebook creator4

• QR code label-printing view

• Linked open data import for vocabularies

• Dynamic notebook-level internationalisation

• Record duplication/templating

• Field descriptions/html infoboxes that are part of the field metadata button

• QR code based photocard

• Templated inheritance of sets of fields in the notebook creator

• Derived/calculated fields

• In-app audio/video recording

• Sliders

• Bluetooth label printers

• Citizen science security improvements

All of the above features are “on our roadmap” as we start moving towards sustainability next year. What do you think we’ve prioritised incorrectly or omitted?

Leave a comment

iOS workarounds — and “First class” and “Second class” apps

First, what do I mean when I say “first class?”5 A first class app “experience” (regardless of technology) is one where it flows “along the grain” of the operating system: everything “just works” without need for other apps, exceptions, or fiddling settings. There is, obviously, nuance here. While providing:

feels like the start of a “first-class” release: using the platform’s app store, has smooth integrated app permissions with the UI, and has an automatic update flow, we’re still some distance off the full experience (mostly due to manual user provisioning and management, and the external GPS flow in the next section). Still, someone (you) could follow the above link, then visit https://forms.gle/yUjnstZZ7EPWWcGz7 and have Android app access to our demo server at https://demo.3.faims.edu.au without undue fuss or instruction-provision.

The stewards barring the staircase6 to first-class offer workarounds like “side-loading” or “Progressive Web Applications (PWA)” where users can get an experience almost, but not entirely, unlike the normal app store. This experience inevitably includes settings menus and security workarounds that normal apps never need encounter.

Our focus has not been on iOS during the last development cycle due to the priorities we had to meet. Despite that, we are very aware of how important first-class (that is to say “published as an App on the Apple App Store”) support is to be a credible cross-platform tool.

Having FAIMS 3 as a second-class “app” on iOS is possible, but with caveats and security implications. (We’re looking for iOS beta testers who can help us explore the limits of this workaround. If you’re an iOS user and are keen to get involved contact info@faims.edu.au and we’ll be happy to give you instructions.)

Adventures in External GPS

On the Android side of the fence, I’ve been working with a Bad Elf GPS Pro+, to investigate how modern mobile OSes handle external GPSes. Quite frankly, little has changed in the last decade — on iPadOS, I need to use the Bad Elf Pro app to provide NMEA 0183 parsing for the OS. At time of writing, I haven’t established that the app provides geolocation data to Safari (see above) — so right now these instructions are android only.

Happily, Bad Elf’s instructions for Android are excellent. Follow the instructions at https://badelf.freshdesk.com/support/solutions/articles/5000708633-enabling-android-mock-location (and as far as I can tell, the apps they recommend can pair with any relatively modern external GPS unit. I’ve got 3 charging (photo above) but haven’t managed to test them yet. Once paired, having the “Bluetooth GNSS” app running (and having gone through its checklist) and provided the unit about five minutes of good skyview to see satellites, it managed to get a 2.7m accuracy on the top of a MQ building, quite close to the promised 2.5m accuracy of the store page.

While these workarounds are not becoming of “first-class” apps, they allow us to collect external GPS data without spending significant development time (and we have too many things fighting for that time right now). Let us know if you want to commission this feature for next year!

Subscribe now

Stuff I’m reading

• Julia Evans on What happens when you press a key in your terminal

• The horrifying short story Lena via a Charles Stross blog post

• Forensic analysis of the JWST photo

• Patrick McKenzie on the Alchemy of Deposits

• Graydon Saunders on Scale

Dev Update

I’ve popped the release candidate tags on our github repositories and we’ve started regression testing preparatory to release! In this development period we achieved six major goals:

• Notebook access controls

• A conflict resolution interface

• Attachment downloading controls2

• A hierarchical vocabulary picker

• Command-line field editing and record deletion/undeletion

• CSV Export

We’ve also worked on bugfixing, increased development automation, and keeping up with the red queen’s race of Node patches.

Our conflict resolution interface

There comes a point in every collaborative online document where two people try to edit the same thing without synchronising their edits. This tends to happen rather more frequently when a product is designed to work offline. In this example, I induced a conflict by opening the same record in two browser profiles (one is called red, one is called purple) and hit save without allowing changes to sync from one browser to the other3.

While the usual case is that these two people will have edited different fields (and thus have their changes merged automagically…) there will be times in any record’s life where the same field has been edited by different people at the same time.4

Upon opening the conflicted record, I see:

A rather strongly worded: “Pretty please resolve your conflict!” (Which doesn’t stop a determined user from editing anyways if they really need to add data.) Clicking on “Why am I seeing this?” explains further:

Conflicts generally arise when two users have changed the same fields in a record, or if the user deleted a file while another user was modifying it. In these cases, FAIMS cannot automatically determine what is correct.

The record will be marked as conflicted. It is then the users' responsibility to resolve the conflict.

Users may continue to edit records whilst conflicts exist, but should be aware that doing so may create further conflicts, it is advisable to resolve all conflicts before editing.

Users can then navigate to our merge interface:

In this interface, they can choose which field to keep, and gradually resolve conflicts until one single record leaves. Unconflicted.

Hierarchical vocabulary picker

Given a vocabulary: https://vocabs.ardc.edu.au/viewById/313, I was able to render it into JSON5 and produce:

A picklist of arbitrary depth, allowing people to specify detailed controlled vocabularies without having a single dropdown 100-items long. It’s neat! We plan to implement our FAIMS 2-style picture gallery and hierarchical picture galleries as part of our August-November development period.

Exporters

We created a command line tool to allow for record editing and “deletion”6.

(The most interesting output is the list command, so you get to see that). This allows for external systems to interact with FAIMS notebooks and make changes to them. While in principle, this mechanism will support record creation … we’re not there yet.

I also wrote a generalised exporter that exports forms in CSV, json, and xlsx formats, the whole database in jsonlines format, and renames attachments to their record’s HRID. Each geometry also gets exported in its own geojson file as well as being embedded in the tabular data as WKT:

Now, folks can actually use their collected data. What a thought!

Leave a comment

Where we’re at and where we’re going

The FAIMS team is now hard at work prioritising7 our August-November development season. This period will bring us to production ready with the ability to support researchers around the world in making notebooks, collecting data, and getting their well structured data out into a useful-to-them state. We’re also working up our sustainability plans for 2023, so if you know you’re going to want a notebook for yourself, your team, and8 your enterprise, let us know!

What I’m reading

• Katy DeCorah, “In defense of magic” — on bridging the gaps between have and need.

• Charity.wtf, “Advice For Engineering Managers Who Want To Climb The Ladder”

• Paul E. Smaldino and Richard McElreath, “The natural selection of bad science”

• Rands (Michael Lopp), “By design” — the necessity of clear communication in leadership

• Mia Sato “The archive saving home sewing history from the trash” — a fascinating physical archive of sewing patterns

• Linus Akesson “Partita Prelude” — a lovely performance of Bach on a commodore64

• “Seafarer” — a little novella set in the “Human Altered” ‘verse.

Subscribe now

2

Not much to show about these. By default, other folks’ downloads no longer sync onto your device until a toggle is enabled:

Sunsetting FAIMS 2.6 — the last in-field observations

Using FAIMS 2.6 has been an amazing experience. The team at FAIMS have been so gracious with their time helping me and my team from the Freshwater Ecology and Ecotoxicology Lab at Macquarie University to develop a module specific to our field needs. The team at FAIMS have been incredibly patient throughout the process of module development, testing, editing the module, testing and editing again and again until we got the module just right!! I’ll admit I was a little fussy developing our final module but for me it was well worth the extra work. I am incredibly grateful as the module worked seamlessly on our recent two-week field trip sampling aquifers throughout the Macquarie, Gwydir and Namoi river catchments. I’ve just done the data download and its all there (yippee!!) photos included and all backed up (probably the most important part!). The process is so simple and user friendly, I honestly can’t believe I use to do all that data entry converting our old paper field sheets into excel at the end of every trip and now all I have to do is hit download! I honestly couldn’t recommend FAIMS and their team highly enough, if you’ve been considering switching to an electronic data entry system the time saved by not having to do all the post trip data entry really just sells itself!

Some of my favourite functions on our module are the GPS and time grab buttons: as long as your device has location abilities all you have to do is hit the button and BAM all GPS coordinates, eastings and northings are automatically imported. This function is great for me as I can easily import these into GIS software and create maps of our site locations. The time grab, which works in the same fashion as the GPS button, is also really helpful and I think it was created just for our module. We needed this function because at the site we have water level loggers that have to be removed prior to sampling, having the accurate time stamp of when this occurs allows us to calibrate this time to the change in pressure that the logger records and water level accurately. Lastly, on our latest module we have added this fantastic vegetation tab which allows us to add entries for each tree of interest in the local vicinity of our groundwater bore. This list of trees and all the info associated e.g. height, species and health conditions etc, is located within the site’s field entry. Therefore, it’s all linked and we can know how many and which trees samples were taken from at each groundwater site, and even more importantly it has photo capture abilities so the pictures saved can help identify each specific tree species if we were unsure in the field.

For me our module works incredibly well for our specific field needs, I even have a checklist inbuilt that reminds me if I have collected all the water quality samples that I need (there’s about 6 little bottles I need to fill and I normally forget one…or two). The capabilities of FAIMS seems endless and like previously mentioned so user friendly even my supervisor (who only upgraded his old Nokia to a smart phone about a year ago) can use it! If you decide to give FAIMS a crack I hope your experience is as seamless as mine has been, and my journey with FAIMS has really only just begun….

Rebuilding this module as a FAIMS3 Notebook

It’s now been a few months since I wrote the first half of this blog, and during that time Brian and I have been working together to move our module onto the new FAIMS3 platform. Firstly, WOW what a change, I love the flexibility that FAIMS3 provides. Firstly, it has retained all functionality and ease of data input that FAIMS 2.6 offered. The biggest improvement is that FAIMS3 now allows users to design and edit modules notebooks with ease. It is so functional and easy to develop a module specific to your needs: it’s as simple as selecting the type of function you want e.g. new input box, adding a label and if for example it is some form of measurement changing the configuration to numeric. Also, there is an automatic “helper text” box which allows you to add additional info or instruction to assist the user about the information required. Then you just keep building from there until the module is complete. Again, similar to FAIMS 2.6 I have a number of tabs to organise my data input into sections that flow for my style of field work. 

I love that now my module can continually be updated and improved, each time I have trialled it I have found little tweaks that can make it better and its just so easy to go and fix them up myself. I expect once given a full trial run in the field that again I’ll have updates and the module will just keep getting better and better with use. 

Again, I just want to thank the FAIMS team for helping me develop the module and now bring it up onto the FAIMS 3 platform. It’s been a joy working with you and I look forward to developing more modules in the future. 

Status Update

Work continues on our CSIRO (A&F)-sponsored features as we get closer to our delivery date. The two major feature that have wrapped up1 are our access/visibility controls and the ability to disable downloads of server-side images. We’ve modelled a UI for record-conflict resolution and are currently developing that necessary functionality.

We are also supporting CSIRO Mineral Resources taking FAIMS3 into the field this weekend. This first collaboration with CSIRO started with a quick proof of concept in 20142 and now they are our first beta testers for the ARDC sponsored side of the project. With any luck, we’ll have photos and video from their first experiences with FAIMS3 in the wild soon!

Our field-ready set of notebooks continues to grow, with a Groundwater sampling for MQ prepared and a port of the Blue Mountains survey (as Penny described on our last blog). Slightly earlier in development, Adela Sobotkova has used the notebook creator to prepare a first draft of the TRAP Burial Mounds 2.6 module migration into FAIMS3 notebook form. The rate of increase of notebooks is increasing!3

Record Visibility & Access Controls

While the ability to prevent huge downloads when colleagues are happily snapping away documenting features is excellent, that feature requires little comment or explanation.4

However, with this new feature, we mark another important step beyond our old FAIMS 2 lineage. In the olden days, all users on a server would be able to see all modules notebooks on a server, and every record within each of those notebooks. Access control and security were designed around airgaps and the pragmatic realities of fieldwork in small teams. If a team needed isolation, they would be given their own server-in-a-box:

Issuing multiple servers-inna-box, unfortunately, doesn’t scale to more complicated projects. (We hope to provide far simpler photos of a FAIMS3 offline server setup soon… soon…) Instead, we have designed multiple roles for visibility and access control on a FAIMS3 server:

• Administrator

• Supervisor/Notebook Administrator

• User

A server administrator will have unlimited read/write access to all notebooks on a server and will be able to see all notebooks on a server. This is the old FAIMS 2 experience. However, for projects and servers that need more nuanced access, we present the roles of “Supervisor/Notebook Administrator” and “User/Team member.”

A supervisor/Notebook Admin will be able to read and write to all records within a certain notebook. But having a notebook administrator role will not provide them the ability to see other notebooks on a server. In this fashion, large enterprises can enforce notebook isolation between various teams, while having a large-scale auditing capability across the enterprise.

A User/Team member will have specific-notebook-only access. In their notebook, they will only be able to see and write to records they, themselves, have created. In this fashion, volunteers, subcontractors, or undergrads can have reduced distractions while ensuring that each user can view only their own records. While this user-mode isn’t for everyone—this record level visibility control is essential for some larger projects and is part of our road to citizen science!5

Reflections on a Common Workflow Language

Crusoe et al recently wrote in the Communications of the ACM: Methods Included, an article on the Common Workflow Language (CWL). In their words, the CWL is:

a pragmatic set of standards for describing and sharing computational workflows. …

In many domains, workflows include diverse analysis components, written in multiple, different computer languages by both end users and third parties. Such polylingual and multi-party workflows are already common or dominant in data-intensive fields, such as bioinformatics, image analysis, and radio astronomy. We envision they could bring important benefits to many other domains.

To thread data through analysis tools, domain experts such as bioinformaticians use specialized command-line interfaces,12,31 while experts in other domains use proprietary, customized frameworks.2,5 Workflow engines also help with efficiently managing the resources used to run scientific workloads.7,10

The workflow approach helps compose an entire application of these command-line analysis tools: Developers build graphical or textual descriptions of how to run these command-line tools, and scientists and engineers connect their inputs and outputs so that the data flows through.

I.e. one the researchers have the data (a problem which FAIMS3 ably solves) — they now need to clean, process, and refine the data. To be clear, the CWL is designed for “Big Data”6 produced by large and laboratory based machines, rather than the “Small Data”7 regimes which so characterise archaeology, ecology, and other fieldwork-based disciplines. However, one thing that Shawn and I are pushing for in our drive to bring preregistration to archaeology, is that data cleaning and analysis can begin before the spade touches the ground.

FAIMS 3 - Electronic Field Notebooks

Dev Diary 6: A philosophy of Digital Archaeological Tools

Brian gives a dev update for mid April, and descends further into the philosophy of technology and archaeology. (Madness intensifies…) Dev update We have just begun a new tranche of development work for CSIRO, focusing on notebook authentication, row visibility controls, new work on the exporter, and atta…

Read more

4 years ago · Brian Ballsun-Stanton

While the current goal is to encourage the use of scripts and small programs instead of manual cleaning and analysis in Excel, our vision has always been towards the ability to contribute large amounts of data which could be integrated into a multi-stage analysis using tools like the CWL. Our push towards open linked data, community standard vocabularies, and shared data collection workflows is fully aimed at larger-scale analysis like the CWL.

Even absent a sophisticated multi-step multi-computer setup like the CWL envisions, the best time to develop an analysis workflow (on whiteboard, paper napkin, or what have you) is right after pilot data is collected. Rather than waiting for full-scale data to come in from fieldwork, developing an analysis suite that can act on your planned data formats and produce useful outputs can provide timely and essential feedback during fieldwork as well as during analysis. The ability to “re-run the analysis” by typing a single-command allows for earlier detection of errors or interesting elements worth attention.

The CWL authors complain:

While (data) standards are commonly adopted and have become expected for funded projects in knowledge representation fields, the same cannot yet be said about workflows and workflow engines.

Nor could it be said about data collection fields. And, to be fair, we are aware of no standards that our XML (FAIMS 2) or JSON (FAIMS3) should be compatible with. But the idea of interoperability of analysis across workflow-platforms should be extensible to interoperability of data collection! One day, it would be amazing if we could use something like the CWL to define a set of inputs which would then thereby create a FAIMS (X) notebook with all necessary hooks to feed the data back into the analysis system. The idea that an analysis package/workflow could generate a way of collecting its necessary inputs is future-facing indeed! Shout out if you know any projects in need of this solution!

What I’m reading

• Greg Wilson on effective abstracts in software papers

• Scott Alexander on DALL-E-2 and stained glass windows

• The woeful security situation of NSW Digital Drivers Licenses

• Jeff Epler’s history of the Story of Mel (one of the fundamental tales of hackerdom)

As we take a breather from focus groups and demos, catch up on some project admin and get reacquainted with our project milestones big and small — all while the hardworking team at AAO keep up the steady march of feature development for the next code release – we’re snatching some time to plan for possible and probable deployments for projects over the next few months.

We know that in a few weeks it will be all-hands-on deck to test, break, debug/enhance new features and start the demos all over again so we’ve found a few hours to start creating and tweaking some FAIMS3 electronic field notebooks—old and new. There’s an industrial heritage survey, a growth monitoring workflow, and not one but two water sampling projects who are back in the field next week and the week after, respectively. Behind that is a backlog of ‘demonstration’ workflows that we are keen to line up and it’s been a nice change of pace to spin up notebooks with our new Notebook Creator, and get stuck into the nitty gritty of whether this attribute should be a dropdown, checkbox, or radio button, that one needs an annotation or certainty flag, and whether we really have a new entity on our hands or not.

Now for the confession: there are few things in my job that I enjoy more than modelling data—any old data—in clear and repeatable ways. There, I said it. It is really quite a delight to grab a paper form, an excel spreadsheet or user guide to an existing workflow—within archaeology or any other field—and wrap, twist and stretch into a new digital form unburdened by what has come before.

The vast majority of my experience wrangling data has been for my own research—ie archaeological artefact data—and often with the constraints of legacy data.1 Modelling other people’s workflow is a delight in comparison (contra the traps of using other people’s data). It comes with the risk of not fulling understanding the researcher’s needs (ie reading their mind) but the advantage is a subtle improvement in understanding one’s own research needs. By drawing parallels in data models and, at times, the simple act of referring to an example from your own workflow to explain a process or a choice for why you modelled a workflow in a particular way, I’m forced to question why I model my own attributes and entities in that way. And, I find myself realising I do it that way because I have always done so. I look forward to the chance to do it ‘right’ next time.2

I’m pleased to say the do-over will be far more gratifying in FAIMS3 than FAIMS2.6 as you can preview your work as you go.3 I’m not the only one to find the Notebook Creator easy to use. My colleague Dr Bec Parkes from Lantern Heritage powered through her first notebook at the focus group earlier this month with very little guidance from us.

As we gear up to model once again ‘the textbook’ single-context recording sheet for archaeological excavation that I once had the pleasure of translating into Spanish some year ago I’m look forwarding to seeing how the paper-digital transformation goes this time round.

“I’m a bit of a numpty, but even I can do this [designing a notebook]” — Focus group participant

Focus group report — Brian’s reflection

The major news this update is that we conducted our first focus group over at UNSW Sydney. We recruited project participants to test an existing notebook and then to try their hands at notebook creation. We brainstormed ideas and improvements and “features we would pay for”1 on a nearby whiteboard as participants worked through sample notebooks and reflected on their experience.

As with any first test of a software product, it was not without hiccups.

After a highly successful recording session out on the central UNSW walk in front of the chemistry building, none of the tablets deployed late last night were syncing. While Penny pivoted to discussion and feedback around the data collection process, I started digging into the problem. Only to discover about an hour later a sync bug due to my fake-users with free email addresses2. There’s nothing like the first deploy in front of a live audience to highlight the gulf of a developer’s “It works for me” and actual pragmatic reality.

We then had a lovely catered lunch of sandwiches3, and turned to the notebook creator. Watching how people engaged with this early-beta software was amazing. They dived right in and started making notebooks and asking about various deployment scenarios. And while Penny and I will miss the flexibility of an entire runtime scripting system allowing for a decade worth of hacks to be accumulated into a working system™, the GUI notebook creator was a hit with our participants. They all had ideas about how they could use FAIMS3 now to improve their various workflows.

Despite teething troubles, our first focus group went very well indeed. FAIMS3 is so much lighter weight (figuratively and literally) than earlier FAIMS deploys: I didn’t have to lug around a server and access point for a local deployment, and the app worked in browser and on tablet! It clearly solves a profound need for sophisticated offline data collection systems which are user-customisable. Now all we need is to finish our iOS deployment4.

Our next task is… to prepare to give this focus group… remotely…

Dev Update

Brian delivered a python script that can edit and delete5 records from an external command-line. We’re making progress towards a proper API! AAO continued working on authentication and notebook and record visibility6. Penny made two new notebooks using the notebook creator as part of our preparation for the focus groups without needing to deploy any code at all. I’m working on this blog post instead of working on our CSV exporter…

Whither Participatory Design?

This reflection stems from work on a forthcoming paper that Shawn Ross and I were furiously editing over the weekend. In that, we were reflecting on what implementations of FAIR data support were adopted by our FAIMS 2.6 users. I observed that there was a large gulf between the interface- and data-desires expressed by those archaeologists with their feet in the dirt and the folks calling for FAIR data7 in 2012-2022. I was also working on a public response to a paper I’m peer reviewing for IACAP, at the invitation of some journal editors, wherein I had cause to refer to Participatory Design.

Participatory design is a remarkably egalitarian technique:

“a process of investigating, understanding, reflecting upon, establishing, developing, and supporting mutual learning between multiple participants in collective ‘reflection-in-action’ (Schön, 1983). The participants typically undertake the two principal roles of users and designers where the designers strive to learn the realities of the users’ situation while the users strive to articulate their desired aims and learn appropriate technological means to obtain them” (Simonsen and Robertson 2012)

Two decades ago, one of the major themes of my Human-Computer Interaction classes was exploring themes of Participatory Design and User-Centered Design. These trends involved the direct participation in users and stakeholders in the design of software systems intended to support them. In many ways, these trends were opposing the Waterfall Software Development Life-Cycle trends in the 1960s through to the 1990s where every element of the interface (if it was ever specified) was designed via committee or fiat — an afterthought to functionality specifications.

These trends led to many many interviews, iterations, and Volvo’s cars. Advice given from CSIRO’s ON Prime incubator echoes the discussion of focusing products on user pain-points to drive adoption. There are calls for slower development, involving users in each stage, and being thoughtful about deployment. These intentions are why we’re running focus groups to get stakeholder feedback. Decades in, they’ve merged into this “Well, of course you get feedback!” impulse.

And yet there’s a tension between users. First, if you ask a room full of archaeologists what they want:

We have found they have trouble agreeing on much of anything in terms of fields and specifics of data collection approach. Which users should we have listened to?8

With FAIMS and workflow-specific notebooks we built workflows for almost 70 different teams. None of these teams asked for FAIR data features9. They were, quite correctly, concerned with the pragmatic realities of collecting data while hot/cold/wet/muddy/student in order to support their research. All of the effort we put into designing GIS functionality went almost entirely unused10.

As a point for discussion this week11 what is a useful balance between high quality outputs, theoretical imperatives, the designs requested by user, and budget? Given an infinite budget, we can certainly achieve a maximal outcome for all persons and all people. Participatory design would push the nomogram away from FAIR data purely into those features desired by users. We’d quite prefer to push (just a little) away from “everything in the same spreadsheet with colours providing meaningful metadata.” Norman observes:

HCD asserts as a basic tenet that technology adapts to the person. In ACD, we admit that much of human behavior can be thought of as an adaptation to the powers and limitations of technology … People do adapt to technology. It changes social and family structure. It changes our lives. Activity-centered design not only understands this, but might very well exploit it.

Where should the balance fall for FAIMS3? What features should we sacrifice for that balance?

Bonus question on the economics of Free, Perfect, Instant

My student Ewan Coopey just linked David Lewis’s recent publication Digital Publishing’s Four Challenges to me. While it avoids talking about data preservation as it is focused on the challenges of digital on publication, the bit on “free, perfect, instant” spoke to me:

That is, a copy can be instantaneously delivered anywhere in the world, a copy is the same as the original, and a copy can be made at zero marginal cost. They go on to explain, “Free, perfect, and instant make a powerful combination, worth more than each of these characteristics separately. Thus, it is very difficult to compete with. . . . For most of history, few, if any, goods and services have been free, perfect, and instant. But with digital, networked goods, these three properties are automatic.”

From the economic perspective, the efficient price for a good, that is, the price that will generate the most societal benefit, is its marginal cost. For a digital good, that cost is zero.

When researchers are willing to pay (relatively) little in terms of time and money for a skeuomorphic representation of their traditional recording sheets in order to get a free, perfect, instant digital copy — but are willing to pay less for a more inconvenient form which other folk might try to persuade them is better in the long term (FAIR data, normalised data…) — how do we square this circle? At present, researchers working with us get what they want because they pay for it. While we try to nudge here and there with advice while we’ve been making bespoke 2.6 modules, even that will fall away with the democratic notebook creator of FAIMS3. How do we balance these priorities such that the platform continues to function, we support FAIR data, and — most importantly — we support researchers in the field?

What I’m reading this week

• A journal article on Digital Publishing’s Four Challenges

• A lot of stuff on the Australian Election12. Here’s a great video on preferential voting.

• Greg Wilson on Role Clarity deficiencies wrecking teams

• LPP Fusion’s quarterly report on the engineering hurdles they’re working past on their way to fusion.

• Morgan Housel on Failures to forecast the future because of surprises.

• And a whole bunch of fanfic and web serials that I won’t share without invitation. Some $unit_of_time, one needs candy floss for the brain.

Dev update

We have just begun a new tranche of development work for CSIRO, focusing on notebook authentication, row visibility controls, new work on the exporter, and attachment sync management. We’re also getting ready for focus groups, spinning up a demo server, and starting to plan to plan our next ARDC development cycle1.

In this new (and un-planned for) development period, we’ve started by paying off some of our technical debt and finally accepting a bunch of dependabot pull requests. We brought our Capacitor2 version up a major version to 3 and are now whacking bugs in reaction to us paying off technical debt3.

While there are still plenty of rough edges, we have a version of FAIMS3 in production, and in real-world use by the people who will be getting their boots dirty. Progress! Ad Astra per Aspera!

On Kearns’ what we lose when we’re plugged in qua archaeotech

In response to Lost Connection: what we lose when we’re plugged in, Kearns quotes Harroway in (correctly) noting that we are all already cyborgs. Just because our phones are not physically implanted in us does not mean that our understanding of what it means to be us, right now, is not fundamentally mediated by us+our personal technology.

The passage that motivated me to reflect on this article for this newsletter is:

Moreover, Vindenes goes on to say that “if we only interpret the world as potential means to an end of ours, we don’t really see the world as it is, or as it reveals itself.” Beyond the distinction of good and good for, technology, if left unchecked alters our experience of reality to such a degree that we’ll only truly experience reality mediated through machines and apps. The real reality of nature could be so obscured that it’ll be foreign to future generations. 

While there are obvious connections here with Slow Archaeology (more on that later), and Heideggerian asides about the standing reserve4 there is a tension here of “mediation.” How does an app, designed to record structured data for purposes of fieldwork, change our experience of reality?

From my perspective, my personal default definition of technology is: “A technology is a tool plus a technique applied to the world to achieve a goal.” (At this point, I no longer remember who I got that definition from.) Notebooks, Microsoft Excel, pencils, apps, phones, cameras, and brushes are all technologies. We normally elide, however, the human-required technique needed to bring technology-into-world for a goal. Intentionality is not required. There may be quite a difference between whose goal is being enacted and who is doing the action — and I believe that this disjunction between actor and purpose is being exploited by virtual reality and the “app” ecosystem and is the thing which Kearns is reacting to.

She says:

While the use of tools has anthropologically defined the human species, technology (especially digital frameworks that can run on their own like AI or machine learning) is in itself a separated and related actor. Technology controls us as much as we control it. 

FAIMS (2.6 and 3) certainly acts as a mediating layer by virtue of presenting/inviting/constraining structured data. By presenting forms arranged in such a fashion, and entities related in such a way — the designer of each module/notebook5 imposes their vision of the reality-we-could-see onto the reality-that-was. A technique (recording method) plus tool (app) to achieve a goal: documenting old occupations. Changing the tool changes the externalities of the goal: who can use the data, the methods of analysis the data is amenable to, and the hierarchies of authority, prestige, and experience which result in literally digging knowledge from the dirt. I would assert, however, that other tools also act as such a mediating layer — and that folk need to make well intentioned decisions about their tool use to achieve their goals.

She concludes:

There is something unique and beautiful about our ability to transcend ourselves and become cyborgs, to create a communication system out of waves and silicon, to at least trick our brains into minimizing the distance between each other when the world is big and lockdowns (driven by pandemics, politics, and war) are real. The nuance and balance that we strike in trying to attain our various socio-political-economic needs is by no means easy, but it’s certainly not found at the extremes of primitively off-gridding or completely plugging into the digital world, however tempting they both are depending on the moment. 

We can think of the same questions-of-balance applying to archaeological fieldwork. There is the unremitting hegemony of time — a field season is only so long. There are budget and political and technological expertise questions. There are local (and university) regulations. There are the comfortable defaults of old fieldwork and the known rewards of a write-once read-onceish publication6. Questions of “balancing” all of these competing needs have only local answers. My only note is to beware of default and unconsidered answers.

Slow Archaeology, an aside

Shawn and I discuss apps and slow archaeology in: Introducing Preregistration of Research Design to Archaeology, which will be available in dead-tree form in early July! Go get your libraries to get Digital Heritage And Archaeology In Practice Data, Ethics, And Professionalism, edited by Watrell and Goldstein!

Kearns notes:

In some corners of modern society, we see a backlash to this hyper digitized movement. Minimalism, off-grid living, returning to app-less flip-phones, hipster record players instead of streaming services, and “slow-living” in general are all trends that (while highly commodified) are on the rise. Slow living itself is a mode and process to try and repair and restore our connection to the world and see beyond the infrastructures of technology and society.

And we make the point in our paper that Slow Archaeology is entirely compatible with good and thoughtful data collection practices, digitally enabled or not. Rather that preregistration and the entire requirement of formalising data quality and structure is antithetical to the “I’ll bang out a data collection form in excel while on the flight out to the dig site” mentality. It is prior intent and consideration of how tools and techniques exist to achieve goals which divides our camps — not the presence of an app.

What I’m reading

• The amazing Julia Evans has a list of newish commandline tools

• Jonathan Haidt’s article on the stupidity of the last ten years echos many themes of my recent research into far-right extremism and conspiratorial thinking online.7

• Most emphatically DeForest White and none of his plagarisers: Plagiarism, and how I'd like to enjoy it

• A transcript of Sabine Hossenfelder talking about wireless power’s challenges.

• Greg Wilson’s review of “Do you really code?”

Much of the conversation concerned communicating across boundaries between disciplines and communities of practice, the research information infrastructure environment in Australia, my changing views on research data and data management, and finally the origins and history of the FAIMS Project itself. Rory released the interview on 30 March:

I hope to dedicate future blog posts to expanding on a couple of the topics we discussed, but here I would like to talk about something that Rory and I touched on only briefly in the interview: why it is important to develop field data recording systems that promote Findable, Accessible, Interoperable, and Reusable (FAIR) data in the field.

Last year, several members of the FAIMS team joined a paper in the Journal of Field Archaeology that summarised some reasons for making data FAIRer during fieldwork, as part of an in-depth review of two FAIMS Mobile deployments at the Perachora Penninsula Archaeology Project (PPAP) in Greece. More recently, we’ve started work on a new paper offering a more focused and thorough treatment of the subject (more on that below).

Publishing research data is essential if outsiders are going to evaluate its arguments, claims, and outcomes. Such evaluation is critical, since it has become clear that much, if not most, research findings are false (see for example research assessments from 2005, 2011, 2014, 2015, 2017, and 2021). Although most large-scale analyses of the reliability of research have been undertaken in laboratory or clinical sciences, there is no reason to believe that the situation is any better in social sciences, or other disciplines for that matter. The situation is dire enough that it is now widely known as the ‘reproducibility crisis’ or ‘replication crisis’ — although in the case of disciplines that regularly employ inductive research that seeks to produce rather than prove a hypothesis, it is probably better to speak in terms of research ‘transparency’ instead of reproducibility or replication (noting important work done by Ben Marwick on computational reproducibility in archaeology).

The underlying data is necessary, if not sufficient, to assess the quality of research findings. To use that data to evaluate research, however, you have to be able to find it, understand it, re-analyse it, and otherwise manipulate it. That’s where FAIR comes in. The FAIR principles encourage researchers to:

• Describe datasets well enough they can be discovered via manual and automated searches (provided they are placed in appropriate repositories),

• Publish data in a way that allows it to be read by both humans and machines,

• Licence data so that reuse is allowed, and

• Facilitate understanding of data by documenting it with ‘metadata’ (data about the data) and providing a clear story of the origin and history of the data (‘provenance’).

It is much easier to produce FAIR data if FAIRness is ‘built in’ from the beginning, rather than ‘bolted on’ at the end. During fieldwork in particular, it is important for metadata to be produced alongside the data and attached to it. If, for example, the location, environmental context, field conditions, time, date, research method, and record author associated with the collection of a soil sample or the recording of artefacts during archaeological surface survey are not recorded when the data is created, at least some of this information will probably be lost. Without it, the associated data is much less valuable.

Pushing the creation of FAIR data earlier in the research lifecycle to avoid the loss of context is a priority for FAIMS. Both the previous and forthcoming versions of the software incorporate features supporting FAIR elements, like the early application of unique identifiers to records, the automatic or manual (and validated) collection of key metadata, the application of explicit data models (a byproduct of producing a FAIMS customisation), use of controlled vocabularies, and cross-referencing within and between datasets.

Between the podcast I did with Rory, and working on a ‘FAIR in the Field’ paper for a special issue of the Journal of Computing Applications in Archaeology (‘Computer applications and quantitative methods in Australasian archaeology’ - stay tuned for updates!), I’ve been given the opportunity to reflect on what we’ve learned from working with dozens of projects in so many disciplines across the past 10 years(!) of FAIMS development and deployments. I’m looking forward to integrating that experience into FAIMS 3.0 so that researchers can create FAIRer data more easily during fieldwork, and pushing the ‘FAIR in the Field’ envelope further as we deploy the new system.

Status Update

The Pilot Soil Monitoring and Incentives Program launched last week and the code that FAIMS3 is working on is powering the app being used to collect well-integrated geospatial and lab data!

In development news, we’re starting to work on updating Capacitor to version 3 for iOS support and planning Q2 development.

On the Colour of Bits, Paranoia, and implications to data collection paradigms:

From “What colour are your bits?”:

In intellectual property and some other fields we're very interested in information, data, artistic works, a whole lot of things that I'll summarize with the term "bits".  Bits are all the things you can (at least in principle) represent with binary ones and zeroes.  And very much of intellectual property law comes down to rules regarding intangible attributes of bits – Who created the bits?  Where did they come from?  Where are they going?  Are they copies of other bits?  Those questions are perhaps answerable by "metadata", but metadata suggests to me additional bits attached to the bits in question, and I'd like to emphasize that I'm talking here about something that is not properly captured by bits at all and actually cannot be, ever.  Let's call it "Colour", because it turns out to behave a lot like the colour-coded security clearances of the Paranoia universe. [emphasis mine]

It turns out that there academic colours as well as intellectual property colours2. In Lakatos’ Science and Pseudoscience, he talks about Research Programmes having a ‘hard core:’

Newtonian science, for instance, is not simply a set of four conjectures – the three laws of mechanics and the law of gravitation. These four laws constitute only the ‘hard core’ of the Newtonian programme. But this hard core is tenaciously protected from refutation by a vast ‘protective belt’ of auxiliary hypotheses. And, even more importantly, the research programme also has a ‘heuristic’, that is, a powerful problem-solving machinery, which, with the help of sophisticated mathematical techniques, digests anomalies and even turns them into positive evidence.

Sitting somewhere between a programme’s heuristic and its hard core are the allowed academic colours of bits: a fundamental understanding of what digitally collected data can and should be able to measure/record/document3. We’ve been down this road before as we negotiate the names-of-things and the organisation-of-things; this feeling is a very familiar one, where the nature of what-should-be is discussed.

Colours in fieldwork

To be clear, my perspective on the colour of bits of astronomers comes from a very very very outsider’s perspective — all I have seen (besides an undergrad astronomy unit at RIT two decades ago) are the expressed priorities in design of a scientific tool by our colleagues at AAO. From my digital humanities perspective, the ‘colour’ of bits in astronomy — their coded-in design assumptions about “what makes a valid record” — tends much more towards the colour-blind CS implementation: a bit is a bit, fungible, and therefore all bits must be held to the same standards of rigour: why would you ever want to commit iffy bits to the database so other people could read them? This design-space of ‘yes, of course you want to share only the highest quality data…’ neglects the pragmatic heuristics of various disciplines I’ve had to deal with.

There, unfortunately, exist many colours of bits in fieldwork-disciplines, due to many reasons:

• Technical literacy and the pragmatic realities of what constitutes academic research over in Arts (and the social sciences)

• Temporal spread of the record (sometimes different people at hugely different times need to contribute to a record before it’s “complete”

• Literal, physical, uncertainty — some things are unknowable from the evidence at hand, but that shouldn’t prevent creating what data we can know

• Granularity — sometimes high data quality isn’t desirable. It’s either unaffordable, not helpful to the goals of the person in charge, not a known problem (we’re trying to fix this last bit), or a judgement call about a rushed field season and a very long data cleaning season

• Divergence of theory and practice.

While I am firmly in camp reproducible data — it turns out that very few of my colleagues are, at least in their expressed preferences in how they want field data collection notebooks created. Aspects of method, discipline, provenance, annotations and certainty, someone’s philosophy of data, and — frankly — what they think a spreadsheet should be used for all colour a bit. We can claim that all of the former are meta- or para-data, but at a certain point it becomes turtles all the way down.

Uncertainty

A decade ago, when I was designing FAIMS 1, I included provision of annotation and uncertainty as part of each field, to allow for a digital simulacra of the scribbling a question mark in the margins of the paper form. It turns out that this was insufficient and too much. A subset of our users would write the ? directly into the textfields where they were recording their data. One especially dedicated user wrote a small essay in an identifier explaining why they made various decisions4. At the same time, the generalised exporter was thought to be a monolithic exporter, and despite 10 years of custom exporters — no one asked me to be able to toggle the presence or absence or different-row-breaking-out of metadata. But all of these examples are bits.

What isn’t describable in “bits” and what I didn’t anticipate, were the strongly varied operationalisations of what “uncertainty” meant. Is it “physical uncertainty to the limits of measurement,” “personal uncertainty due to degradation in thing being observed,” or “Help! I need an adult supervisor!” When I designed uncertainty as a sliding scale between 0 and 1, I was thinking quite strongly of Bayesian reasoning, being influenced by HPMOR and the folks at Less Wrong. (I thought that the ability to document one’s epistemic state while collecting data would be useful.) The problem here is that by compelling uncertainty into a decimal value, I’m not actually sure anyone actually used it the way I designed it. I was trying to take a colour-of-bits problem and stuff it into bits.

In FAIMS3, we’ve turned uncertainty into a checkbox and allowed for a custom label — which is certainly one way of doing things. It allows for people to check, for example: “data taken with handheld GPS.” And while this uncertainty (meta)data is certainly useful — it’s even farther away from documenting the colour of bits than we were when I started.

(Meta)data doesn’t capture the evaluative accent

In many ways, the perceptual colour-of-bits is Voloshinov’s evaluative accent5:

Every ideological sign – the verbal sign included – in coming about through the process of social intercourse, is defined by the social purview of the given time period and the given social group. So far, we have been speaking about the form of the sign as shaped by the forms of social interaction. Now we shall deal with its other aspect – the content of the sign and the evaluative accentuation that accompanies all content.

…

What determines this circle of items endowed with value accents?

In order for any item, from whatever domain of reality it may come, to enter the social purview of the group and elicit ideological semiotic reaction, it must be associated with the vital socioeconomic prerequisites of the particular group’s existence; it must somehow, even if only obliquely, make contact with the bases of the group’s material life.

Individual choice under these circumstances, of course, can have no meaning at all. The sign is a creation between individuals, a creation within a social milieu. Therefore the item in question must first acquire inter-individual significance, and only then can it become an object for sign formation. In other words, only that which has acquires social value can enter the world of ideology, take shape, and establish itself there.

Bits only have meaning as sign-arrangements. Literal, computational, bit-arrangement and parsing misses the point of bits having semiotic weight. Arts scholars, in my personal experience, have an Aristotelian evaluative accent — the colour of a bit is informed by its academic pedigree, not simply the rigour by which it was collected.

Security clearance, trust, academic side-eye, and colour-coding cells of a spreadsheet as data6 are all expressions of this infological evaluative accent. They are a “frame” by which we parse and understand the affordances of the bits being presented to us. What can we do with this data? What should we do with this data?

To some in arts, high quality data with a red security clearance is far less valuable than, for example, blue data shared as a photocopy of some scholar’s table (or pdf in these more modern times) when they worked it out by hand some time ago. To others, data is only an intermediate byproduct — and it can only increase in quality by being passed around and edited. A lack of validation on fields may stem from the fact that the field season’s permit is strictly 6 weeks long and being able to capture any data while in the high-friction environment of standing up to one’s knees in muck is better than fighting with the tablet. (Or, perhaps, acknowledging Rumsfeld’s unknown unknowns and knowing that the specific data collected while in the muck will be quite difficult to anticipate while dry at home).

The presence of non-fungible bits is an affront to those of us who work on “big” datasets7 as it becomes very difficult to compute on these evaluated dimensions of meaning. However, when the time, cost, or pragmatics of the reality allow for either the collection of colour-coded data or no data at all … most folks in Arts would prefer some over none, even if it is hard to describe in a FAIR fashion. If we don’t support them, we’ll never be able to persuade them that tools beyond the word processor and the spreadsheet are suitable to their task. This judgement is a mistake.

Reading this week

Julia Evans on the importance of tiny learning milestones while programming

Patrick McKenzie on virtual sword accounting (where I found the above article linked) has amazing quotes like:

And that is why accounting standards have destroyed more imaginary swords than all the rust monsters in all the prime material planes combined.

Huemer on What is Magic?8

Andrew Hudson in Ars Technical provides A brief tour of the PDP-11

Fond memories of Paranoia. There will be music, dancing, happiness, all at the carnival, By order. Are you happy, citizen?

The problem with manic deadlines in lieu of leave is that they leave very little capability for “normal work.” Updating the blog, unfortunately, is that sort of “normal work” — important outreach, but always second priority.

While we’re expecting a public announcement soon, I’m going to have to be somewhat cagey in discussing why we have unexpectedly had a 6 weeks of extra development. But that… belongs in the status update!

Status Update

The status quo ante1, when we wrote dev diary 3, was that we were going to start an ARDC-funded development period roughly a week ago, and would continue for about ten weeks. That development period would bring us to public beta and we would be running focus groups2 during this time to start accumulating user feedback. The team would have taken a month of leave in December/January and we would be refreshed and ready to face 2022.

Instead, one of our long-term contacts came to us with what can only be described as an incredibly urgent opportunity. If we were able to deliver a functional notebook for a major project — they would fund the rapid acceleration of our dev program.

As such, rather than the thinnest possible bridge between a theoretical notebook creation and export, we now have a server with dedicated infrastructure hosting 21 instances of FAIMS3 (to support 20 different groups and testing) each compiled to Google Play with a robust field manual and a well-tested exporter to jsonl. The 6 weeks of panicked development got us server-level authentication (using DataCentral as an example authentication service), many many sync improvements, and many many many bugfixes.

We have shipped FAIMS3 to prod(ish)!

While there are still many features on our roadmap, such that we’re not comfortable offering our electronic field notebook as a generalised software as a service — we are able to offer concierged notebook experiences to those willing to live with few bells and whistles. Hence, FAIMS3 is “in production”-ish. We’re going to resume ARDC-sponsored development shortly, but this was an amazing opportunity to concentrate on fit and polish to deliver something specifically useful for a nation-wide data-sampling campaign.

We’re all also busy playing catchup on our other “normal work”, so more details later.

Thinking About Technical Debt

When Martin Fowler wrote about the challenges of dealing with Technical Debt during scaleup it immediately put me in mind of the developmental challenges we faced during January. As they kept articulating example after example, I felt that our current situation felt awfully similar.

This is not a problem.

Technical debt is the accumulated externalities of decisions made under resource constraints: time, money, attention, or material. Tradeoffs happen and usually the decisions are made intentionally or are forced as a function of the larger operating environment. I certainly see how decisions I’ve made around the planning and implementation of FAIMS3 have expressed elements from Company A (ship it fast!) and from Company B (Do everything right from the start!).

Instead, I want to muse briefly on the fortune/blessing/providence/circumstances that forced our hand to ship to production much earlier than expected. The demands when shipping code to customers with their boots on, ready to step into the field, are very different than those when we are building a product appropriate for an MVP (minimum viable product).

A persuadable demo is heavy on features or capability. By insisting on a notebook generator and on round-trip capability, I was sacrificing both notebook-specific features and a deep exploration into increasing reliability in the interests of being able to tell a compelling story. My plan was to have enough functionality to be able to demonstrate our product in carefully controlled focus groups that would then be able to have good discussions as to routes forward and unforeseen design consequences.

Instead, by changing our focus to a need to ship to prod now! now! now!, there has been a focus on data scale, server automation, and the pragmatics of authentication and sync. These features, while necessary, are not headline features during a demo — but they are far more fundamental for the day-to-day data collection needs. Thus, a large chunk of development was the unexpected facing of technical debt, rather than the planned and smooth reduction of debt possible when external deadlines were not screaming at us. At the end of the day, however — the urgency of shipping to prod focused our minds wonderfully and we were able to achieve a pilot product accepted by the clients.

While there were certainly things I would have quite grumpily communicated to my past self — how could I have been so stupid as to focus on [redacted] — we had relatively little debt to pay off and were able to do so. While we haven’t even entered the exponential elbow of scaleup to more than one client yet, I’m far less concerned that we will be unable to do so.

Thanks mystery client! By forcing our hand to ship to prod very early, we’ve paid off our technical debt far earlier than expected! This was very useful, even if it was a little frantic!

I’m not sure what my advice would be to other projects who are also facing a runway to shipping their MVP. The continual advice that we got was to ship early. Our project is so complex that I think the 30-odd weeks of dev was the earliest we could have credibly shipped the project. I think my advice is ‘Get someone to pay you to ship to prod when it is uncomfortable but not impossible for you to do so.’ Good luck implementing that advice!

Reading this $unit_of_time

Somehow the $unit_of_time which defaults to ‘biweekly’ is now… ‘quarter’ — I think I need to file a bug report.

• Naomi Kritzer writes an excellent little short story on The Dragon Project

• Simon Willison shares an interactive explainer of SHA256 by Domingo Martin.

• Charity shares some useful “bad smells” to be really concerned about during an interview.

• Wouter writes about different sorts of backups — an excellent explainer to pass onto your colleagues.

• Chris Lee reflects on the challenges of applying a new curriculum.

• Matt Blaze unpacks3 The Cryptography of Orphan Annie and Captain Midnight

• danah boyd4 both uses text emotes like I do ::cheers:: and has transcribed one of their lectures about responsible data science.

• Marianne Bellotti shows how organisational hierarchies make certain sorts of technical debt extremely predictable.

• Filippo Valsorda provides a strongly worded wakeup call around professional maintenance in the open source community.

• Cedric Chin discusses tacit knowledge versus deliberate practice.

• Amy Shira Teitel researches alternate lunar rovers from the 1960s! Futures that could have been!

While my first dev diary has our expensive sentence, the way I framed our goals when talking about feature priority was, “the thinnest possible bridge between notebook creation and export.” We need to be able to run demos of this software in the new year, and by being able to take people on a journey from notebook creation, to data collection, to (maybe) round trip, to export — we can exercise all aspects of the system. This lets us avoid the trap of “too many features” in our data collection while still requiring manual design of data collection steps. That bridge also would need to try to support our CSIRO Geochemistry water sampling workflow. I had our devs focus on water sampling because of the multiple pH/Eh samples required in that workflow. The rest of the module is implied if we get water sampling right.

To a first approximation we achieved this goal. Follow me on the journey across this incredibly narrow bridge.

Hark! It says… "Create Notebook!”

On 8 December 2021 I built a simple notebook without help from our devs and without need for manual intervention. I was able to specify metadata, form and section ordering and fields, annotation and certainty labels and got a bunch of JSON. Once loaded onto CouchDB1 I loaded the app in Chrome and saw:

A short setting the auto-incrementing2 values and new record later, and:

The form and section and fields and field metadata I specified was there for (me) to see! It worked! After making sure the data synced, I was able to run my exporter against the CouchDB instance and see:

A folder with a CSV in it that had my data in the csv! (It’s very important to check that final step.) While there is plenty of distance to go on export3 — I was able to see the data that I had entered on the server in the form that I had designed by GUI! THINNEST POSSIBLE BRIDGE, ACHIEVED!

Some other neat stuff

We encode a visual model of form relationships (and also implement them) so that both containing and linking relationships are supported and can be specified using the GUI4. We also have map widgets from Steve, an entire external Authentication feature-set, merge-conflict detection5, multiple-sections, and human-readable identifier search. All of this was developed during the extremely miserable Sydney lockdown and our developers over at AAO deserve a big round of applause.

Reach out to us at enquiries@faims.edu.au if you want to be involved in our private demos early in 2022.

User Acceptance Test Reports

After a nice long chat with Nathan Reid of CSIRO Mineral Resources, we demoed all of the work that AAO had put in this development season. He was able to use the notebook functionality, export data, and even create a notebook. While there were plenty of bugs (most of which we already knew about) — there were no showstopping problems and everyone agrees that we will be able to use FAIMS3 to collect data in the field sometime next year.

Which is nice.

With this v0.3.0 User Acceptance Test success, we’re calling 20216 to a close for development work and FAIMS3. We already have way too much planned for January, but we’ll release news of that in the new year. 2022 will be a big one.

Reading this week

(Insert appropriate exhausted look here — but happily I’ve got Newsblur as my RSS reader and can just look at what stories I shared in the last two weeks)

• Patrick McKenzie on The Fraud Supply Chain

• Scharon Harding’s writeup of someone restoring a keyboard and trackball from an old ICBM control panel.

• Rachel on ways a (code) builder culture can fail

• Apenwarr on sociotechnical patterns from our current software development trends.

• Brian Krebs on how the net is held together by spit and bailing wire.

• Chiptunes of Sea Shanties (Youtube Playlist, autoplay) by The Longest Johns and Lucy Humphris

FAIMS 2.6 will not die

It is 2021, FAIMS3 is not quite ready and in April I unexpectedly needed to collect data on Cold War bunkers in Aarhus. I decided to reuse an existing FAIMS 2.6 module1. Here is my experience:

There are three different ways one can create or edit a FAIMS 2.6 module fit to one's needs:

1. Write a module definition packet from scratch. 

2. Edit an existing module’s definition files (existing definitions are published on Github).

3. Edit a module instantiated on the FAIMS server via the server web application’s graphic user interface.

Setting aside the first and third approaches for now, I decided to take the second approach. Editing the module’s definition files requires intermediate computational skills, as the module creator needs to locate the existing module on Github, then download and edit the four XML / text files that comprise the definition packet (contained in the module folder on GitHub). In my case, I changed the names of tabs and attributes to correspond to my own project. One needs only find and replace skills, but attention to detail is necessary as well as the audacity to edit XML. 

Knowing this, and being no programmer, I still choose option two when I want to mashup a new module from pieces of existing modules on Github. Once fieldwork is going, I use option 3 to add new elements (maps and vector data) or make urgent in-field changes (like adding forgotten terms to controlled vocabularies).

Aarhus Cold-War Bunkers

In April 2021 as Denmark was coming out of its second lockdown, my colleague Petra Hermankova and I were looking for an outdoors activity to engage Spatial Analytics undergraduate students. We decided to adapt one of the modules designed for citizen science in Bulgaria — burial mound monitoring — for documenting civil defences around Aarhus. Upon arrival to Aarhus two years ago, it struck us how ubiquitous shelters, bunkers, and other military installations are in this urban environment. Their density resembles military installations on the Bulgarian-Turkish-Greek border where we worked in 2018 — yet that is a former Cold-War border zone and now a closely monitored EU border.  We knew little about Cold-War bunkers, but found an article about them by Rosanna Farbøl, a historian who researches Danish civil defense. She noted the lack of spatial information on the civil defenses and so we decided to remedy the situation. I would create a prototype module for documenting the bunkers, one that we could jointly test and then pass on to History or Spatial Analytics students to use for research projects. 

Adapting an existing module

FAIMS Github library has several modules which allow for user self-registration and login and are thus suitable for lightweight citizen-science or self-selected student projects. To document bunkers we needed to collect spatial points and visualise them on a basemap of Aarhus, fill out some structured data attributes (mostly using controlled vocabularies), and attach photos. All of these features were in the burial-mounds-community module and so I chose it as the starting point. 

The adaptation process itself had two stages: 

1. Edit the definition packet to update the entities, plus their attributes and controlled vocab terms, from burial mounds to Cold-War bunkers.

2. Find and prepare basemaps and configure the FAIMS mobile GIS to support our spatial reference system.

Editing the definition packet

In the first stage, I downloaded burial-mounds-community from Github, opened the module folder and edited three files in the definition packet (the data_schema.xml, ui_schema.xml, and properties.txt) using my favorite text editor software. I replaced every instance of 'mound' with 'bunker' or 'shelter', and updated all field names and vocabs to relate to civil defense rather than mortuary archaeology. I also edited html files in the ‘data’ folder which contain the module’s landing page (with project-level metadata) and in-app help text.

These changes took about an hour, because I was figuring out the replacements on the fly, and I had to be absolutely consistent. For example, FAIMS allows separate manipulation of the data schema and user interface (UI), but the two must be kept consisent. When I changed the drop-down list associated with ‘Feature type’ (containing 'bunker', 'shelter', 'depo', and 'trench') in data_schema.xml, I had to make exactly the same changes in the ui_schema.xml, all the while being careful not to introduce any blank spaces or other intrusive characters. Such manual changes are tedious and risky because a lapse of attention will make a change in the data schema not render properly in the UI of the final app.

When I had made some changes, I saved the files and instantiated the module on the public FAIMS server (add Footnote to Instantiation). Then I downloaded the module to my phone and tested it, paying special attention to the items I changed. I continued to download, edit, save, and instantiate the definition files until I was satisfied with the look of the customised app.

It took me two iterations and 1.5 hours to have a workable app with meaningful terminology. There were several things in the original module I did not need, but that was OK (I left them in the app where they could be ignored). What I needed next was the basemaps. 

Editing maps

Anybody who wants to collect and display spatial data faces a second stage of customisation, spatial data preparation. Finding and pre-processing rasters and vectors for use in FAIMS can be done in desktop GIS or on the command line. I downloaded a satellite image of Aarhus from a local provider at kortforsyningen.dk. Processing this 4-band, 1m resolution, 20Gb ortophoto (.ecw) image into nine downsampled geotiffs of ca 100MB each was the most time-consuming part of the whole adaptation.

After the initial splitting of the image in desktop GIS, I followed the raster-processing guidelines to define the projection, tile, and add pyramids to the map files with three one-line commands in git bash2. The mobile GIS engine in FAIMS 2.6 (an older version of Nutiteq, now owned by Carto) is 3D and uses the web-Mercator 3857 reference system. Tiles and pyramids accelerate the display of rasters on mobile devices. I did not need to prepare any vector data, but when I have shapefiles I also project them into EPSG 3857 and bundle them up in a spatialite database following these guidelines. 

After my basemaps were prepared, I viewed them in a GIS viewer to test their rendering. I then put them in a ‘Map’ folder on the server and added this to ‘data’ folder3, where I kept images associated with the app (e.g, ‘picture dictionaries’)4, and zipped them up into an compressed file named data.tar.gz5.

In my next post, I’ll talk about the experience of “spinning up” a module and collecting data in the city. Here is what our points looked like midway through data collection:

I’m writing this watching a timer count past 5 minutes while building FAIMS3 using Github Actions. Our Continuous Integration/Continuous Delivery CI/CD) pipeline that rebuilds our webapp and our native Android and (soon, I hope) iOS apps so we can test that all of the “bits” are working together. This week I added our “conductor” server which handles user authentication and I’m working on automating iOS uploads to the Apple App store.

Status Update

We now support file attachments, and integration with photo-taking capabilities on device! Above is a screenshot of an unscalled photo of some lovely trees that I took while on a ferry ride.

Other features in the last few weeks include:

• UI updates to highlight notebook creation as well as data collection

• Relationships and child-records in the notebook creator!!!!!

• Very very very preliminary work towards a round-trip capability1

• Annotations and certainty support!

• Manual app creation on Apple’s App store (not-public but a necessary precondition for the work I’m doing today).

• Authentication using AAO’s DataCentral — this time we have real authentication which can survive on the internet.

• And some more work on the GIS/Map plugins by Steve!

Everything seems to be happening at once as we enter feature-freeze as all of the various threads of work over the last months get polish, attention, and final commits.

Thinking about a decade of Stack Overflow

When working on all of this DevOps2 a post by Greg Wilson3 kept resonating with me on when reading the answers on stack overflow is not enough:

I spoke with someone earlier this year who had been using the Unix shell for several years but had never used the man command. Whenever they had a question they went to Stack Overflow: experience had taught them that they could find their answer there more quickly than by hunting through comprehensive breadth-first documentation written by people who are guessing what the reader wants to know rather than responding to the actual gaps in their knowledge.

In years past, I was a moderator on the Stack Exchange network4 and a not inconsiderable amount of my moderation time was spent trying to get people to move very useful information from the comments back into answers. People would leave answers, have a fascinating clarifying discussion in the comments and then wander away — job done.

And yet, in this hodgepodge of answers and conversations and "edits signalled in text” (which we frowned upon), is the most amazing resource for software development. And somehow those of us who use this resource successfully know how to interrogate the answers, the answer metadata (when was it answered, by whom, with what rep), the answer paradata (comment threads) and the general tone of activity around the question to judge how to most successfully use it to achieve our goals.

The art of using Stack Overflow successfully is not taught, seldom enforced on the site itself, and essential to advanced modern software development practices. Thinking about technical documentation as a form of professional knowledge transfer, we have moved from closely guarded guild apprenticeships to formal and well-typeset manuals5 to… embedding links to Stack Overflow in error messages instead of links to a manual6.

We live in strange times.

Some time later…

This post has been worked on (while waiting for builds to run and fail and run and fail and run and fail and run and fail and-^C) for a few days. The perfect coda to this post is a screenshot of a very-long-runtime command that finally worked.

Now all I have to do is make this into a pull request to our main branch and we’ll have automatic deployment to Apple’s TestFlight and App Store along with all of our other CI/CD infrastructure.

Reading this week

Outside of extremely fluffy fiction reading, notable stuff from my RSS feed this week is:

• Amy Shira Teitel’s on Jetpacks on the Moon

• Scott Alexander talking about (eventually) the philosophy of science as a function of the discussions around Ivermectin

• Brian Krebs on advances in phishing using sms alerts alerting to bank fraud

• Alex Wellerstein on a decade of running an academic blog

• Rachel on all of us being clueless programmers once

Status Update

Our Expensive Sentence for private beta is:

To support CSIRO Geochem for in-field testing in November 2021 by allowing minimal module design through web interface, collecting complex relational data with multiple devices while offline, and supporting API import and export from/to structured formats like CSV.

As a brief status report:

• I was given a brief demo of our notebook generator on Wednesday. No more hand-writing XML! We even have access control built in from the very beginning!!

• We have relationships and child entities — no need for flat files here.

• Steve has made some amazing demos for the GIS portion of things using OpenLayers

• And I’ve written exporters to export in CSV, geoJSON, KML, Shapefiles, GeoPackage, along with a sample exporter to ResearchSpace.

This makes the thinnest possible bridge across the chasm of designing an electronic field notebook for data collection, to … actually collecting data, to getting the data out of the system. We have a ways to go yet, but progress is quite promising.

Let us know if you want to be involved in focus group testing in the new year.

Thinking about Licenses

Thinking about the recent news of AGPL’d license being run without offering licensed downloads1 and the fact that they aren’t offering source downloads being used as the excuse to sue the site (very Capone-Tax Evasion style) made me interested in what our current license exposure was in FAIMS3. Running npx license-checker --summary on our primary repo2 has produced a bewildering array of licenses:

It’s quite clear that I’m going to need to spend some time with the license compatibility checker (via this post) and to try to figure out if any of these licenses (or any yet to come) are problematic. While our code is public. we’ve chosen an MIT license to hopefully support future sustainability efforts. We will, however, need to be careful not to let any Affero Public License’d stuff be included as upstream dependencies in our codebase.

Why everyone cannot use the CRAPL, I simply do not know.

Reading this week

This week, I’m reading Chapter 1 of Cybernetic Revolutionaries — and thinking about cybernetics (command and control, not robots), dominion, and management.