Better sit thus, and observe thy strange things

I really like the

 file drop

feature of ownCloud. You can configure a directory to receive files from a thrid party without authentication and with no troubles. You get a URL and that is everything you have to share. People will see a simple file upload field and can push their files directly into your ownCloud. Easy.

Upload to ownCloud without auth*

Sometimes the most sophisticated deployment strategies and infrastructure models can unfortunatly not be shifted into the real life, so one has to go back a few decades and use tools from that time and mix them with current ones. In a former post I described a cool solution for building a static HTML website and putting it into a Docker image.

(S)FTP and Gitlab CI

On many of our Websites we have contact forms. And recently there seems to be an elevated number of bots filling out these forms with spam data. That’s especially a problem, when a contact form is directly connected to an Issue Tracker, that automatically generates an issue when a mail to a specified address arrives. A possible solution for avoiding automated form-fills is using CAPTCHAs.

Spam-Protect a Symfony5 Contact-Form with CAPTCHA

Since a couple of years we prepare applications for the eXist-db ecosystem based on the EXPath Package standard. Last week we setup the first test environment. No questions; no excuses: Tests are extremely important! In some cases one can start developing a software by specifying a test, so called Test Driven Development.

A Testscenario for EXPath Packages

Von Athanasios Mazarakis und Paula Bräuer  Eine Teilprojektvorstellung unseres Partners CAU Kiel zum Projekt OA-FWM   Das vom BMBF geförderte Projekt „Fachspezifische OA-Workshops und Bereitstellung von OA-Materialien - OA-FWM“ zielt darauf ab, insbesondere die fachspezifische Information und Ansprache im Bereich Open Access zu verbessern.

Die Wissenschaft retten und dabei Spaß haben

First of all - I could have put much more words in the title because we are covering a complete workflow for our TYPO3 documentation (for our customers and their sites).  We will see the tools used, how it is organized and finally how it is deployed. Format   TYPO3 used to use an OpenOffice-based documentation shipped as *.sxw files ages ago.

Documentation with reStructuredText for TYPO3

When going for Text Mining a collection of resources pretty much the first step is to check for the language. If it is a multi-language collection may be there is at least

 some

metadata. At SUB we often use the METS/MODS standard based on XML.

You might not need Metadata

The Alliance of German Science Organisations has recently published its “Recommendations on the Development, Use and Provision of Research Software”, in English and German. Research software has been an emerging subject in recent months, which has been considered as a subtitle of research data by the scientific community for a long time. But software is not data. It must evolve and adapt to the changes in its environment.

Research Software Sustainability

Am 5. und 6. Dezember fand der „22. International Workshop on Treebanks and Linguistic Theories” statt, auf welchem auch das SUB-Kooperationsprojekt „Edition des Ugaritischen Poetischen Textkorpus” (EUPT) vorgestellt wurde. Doch was sind „Treebanks” (z. Dt. „Baumbanken”) überhaupt und welchen Nutzen haben sie für die Forschung und Entwicklung?

Baumbanken

A few years ago an internal initiative on software quality launched at the Research and Development Department. There is always room for improvements and some guidance can help implement best practices for better workflows and for better code. The very first stop on our ride was at Joel`s Test for better code, published 24 years ago. It is time for a review — again. Do you use source control? Who doesn’t? Can you make a build in one step?

The Joel Test: 12 Steps to Better Code

In dem Podcast „Code for thought” erzählen Kristine Schima-Voigt und Kay Liewald über unsere Erfahrungen mit agilen Methoden in der Gruppe Software- und Serviceentwicklung an der SUB Göttingen. Peter von „Code for thought” wurde auf der „Conference for Research Software Engineering in Germany“, kurz deRSE, auf uns aufmerksam.

Podcast „Code for thought"

Conducting debug sessions   In our teams, we have repeatedly conducted debug sessions to help team members get started with the development. An entry point is pointed out and the control and data flow is followed together. At important code points breakpoints are set and the state (variable values) is inspected. The whole thing can be recorded as a video cast, e.g. with BBB and is available for recapitulation or for others.

How to simplify the familiarization with the code

We are looking for developers who fit into our team - so if you want to work in a great work environment in a beautiful city (Göttingen) and are an experienced software developer, please apply. We are working with a wide range of technologies, so we hope, there’s something for you.

Developers wanted (PHP, Java, TYPO3, Frontend, Symfony, Go, ...)

(Dependency Track, Best Practices) Introduction   In May 2021 President of the USA Joe Biden has issued the Executive Order 14028 on “Improving the Nation’s Cybersecurity”. The document makes several demands on federal agencies concerning the implementation of Cybersecurity and identifies the security and integrity of the software supply chain as one vital claim.

SBOM, anyone?

On the 14th of october a ‘IIIF-Friday’ will be held at Ghent University. Most importantly, the major IIIF-Viewers are being presented, followed by a Q&amp;A session.  Feel free to register for the online event at meemoo.

IIIF Friday at Ghent University

The colleagues from the other side of the library also started blogging, but more of the book- and collection related library topics.  For hosting the blog, the platform hypotheses.org was chosen. Hypotheses.org mainly focuses on academic content.  Check it out at https://sub.hypotheses.org/ or subscribe to the RSS Feed.

New blog from the colleagues

I was quite sceptical if an online conference could make me pay attention in the same way a “physical” conference could. Spoiler: It could. There were some minor issues, such as not so perfect sound, but it was sufficient. The conference took place at an online platform (hopin). All sessions were pre-recorded and made available for all attendees after the conference.

SymfonyWorld Online 2021 Summer Edition

Creating screencasts, for example for documentation, is easy with these three powerful open-source tools: OBS, Kdenlive and ImageMagick, which all run on Linux, MacOs and Windows respectively. After downloading and installing all three, (https://obsproject.com, https://kdenlive.org and https://imagemagick.org) you are good to go. Let’s start with creating the screencasts.

Creating videos for documentation

You may know the dark mode from your mobile phone or elsewhere. It’s basically a theme that can be activated, but also can be a system-wide default. Most browsers do also support this setting - and we as web developers can read these preferences (Firefox) and use media queries to apply a dark theme for our website. Note that this is not togglable, as some JavaScript solutions offer this feature, but only reacting to the browser preference.

Darkmode in CSS

In our University - and also our Data-Center GWDG we use Open-Source Software whenever it is possible. So, instead of using proprietary Video Conferencing Services, such as Zoom, Google Meet or Microsoft Teams / Skype, the GWDG provided us a reliable Big Blue Button Service. This Software is - compared to the proprietary solutions mentioned above - limited in its functionality, especially the fun part!

Fun with Big Blue Button

Last week I attended the official 2019 IIIF Conference in Göttingen. It was maybe the hottest week of the year, so the first thing the IIIF Consortium bought, was a fan to get some fresh air for the audience. IIIF? IIIF (Triple-eye-eff), the

 I

nternational

 I

mage

 I

nteroparability

 F

ramework is a set of APIs for having a standardized retrieval of Images, Metadata and more.

IIIF Conference Göttingen

When writing code at some point we want to highlight a specific state, we want to celebrate reaching a new milestone and usually this is the moment to prepare a

 release

. Nowadays when we all are about to use git, it is quite common to utilize

 tags

for this. A tag points to a specific commit, it is a persistent shortcut to this point of the development.

Gitlab Releases and Blobs

The way we are using TYPO3 has constantly been changed in the last 10 years. I will show you, how we use state-of-the-art technologies to optimize and automatize the way of building and deploying our TYPO3 sites. History   There were times no source code repository existed and everything was done live on the server. Then we had monolithic repositories in Subversion (do you member?) where each and everything belonging to the site was stored.

A modern way to organize and build your TYPO3 Site with Composer and Webpack Encore

Well, at least partially self-updating. You still will have to write the sentences and all that…  Suppose you are at the beginning of a project and have written some classes.

How to write a self-updating documentation

Well, this might be a catchy title. But what we are going to do, is to put the Jekyll artifacts into a Docker image and get a completely self-contained website. Jekyll, what? Artifacts? Ok, let’s start from the very beginning. Jekyll is a static website generator. This blog is run by Jekyll. That means, we do only have to edit markdown files and a ruby program generates everything around that and transforms the posts into a website.

Serving a Jekyll Blog with Docker

Tired of presentations made with a full-featured (as in: full of features you are never going to use) software suite stored in unhandy proprietary formats? OK, I am sorry for this rhetorical question.

Preparing Presentations with Markdown

I just stumbled over a nice post and became inspired writing about agility in our teams, how we got there, how we use it and how it may fail. The intellectual foundation for becoming agile in a software development group, is to internalize the “Manifesto for Agile Software Development ”. Starting the transition   We started off with a new project about 4 years ago and some people that were keen on changing the way we worked in the past.

Agility

C

reate

 R

ead

 U

pdate

 D

elete — these operations describe the most common tasks a relational database is used for. A common task on top of that, is to build interfaces for users, to execute these operations in a nice way and create a frontend for these task.

A Simple CRUD Application with Symfony

Just because you have some geographical data doesn’t mean you should always map it but often the usage of data visualization with an interactive web map application can bring benefits to your project. Map visualization helps to reduce the degree of abstraction and supports communication and sense-making (a.k.a. data analysis) of your data.

Be kind to your users-use a map application

This blog post accompanies my DH-Tech talk about config management.

 tl;dr

: There is a recording available. Problems with infrastructure administration Operating an infrastructure can be hard. Among the most common problems is that of reproducibility, i.e. Is the documentation both current and correct? Is it clear how to do a fresh install or set up a secondary system? Another is that of uniformity, i.e. are there any standards?

Config Management

A Do-A-Thon event took place at the SUB on the 14th and 15th February that brought members from the Göttingen Campus and the Wikidata community together in order to dive into the world of Wikidata and pilot projects. For a research library a knowledge base like Wikidata should be of particular interest.

Wikidata Do-A-Thon

It has been quite a while since the last version of SADE – the TextGrid flavoured version of the »Scalable Architecture for Digital Editions« – was published. Here is a new version, with a new deployment strategy and a complete new architecture. Everything is new in 2018.  Once this version of SADE was derived from a development that Alexander Czmiel (TELOTA at BBAW) promoted in »Editio ex machina - Digital Scholarly Editions out of the Box«.

SADE²⁰¹⁸

Technology fails – at least sometimes. This is particularly true for a modern distributed research infrastructure, such as DARIAH-DE.  For the operation of this infrastructure, we have implemented a monitoring solution with Icinga. This enables us to be informed about problems and react quickly.

Static Infrastructure Status with Jekyll and GitHub Pages

The continuous integration toolchain has developed a lot during the last years. Our mission, to build and deploy everything completely automatic and reproducable became more and more elegant. But let me take you back into the ancient halls of our workflows. Let’s take a TYPO3 CMS website. We built sites with this PHP-based Content Management System for ages, but let’s skip the dark woods of Subversion.

Building and deploying applications

Welcome everyone! This is the place to read about projects, resources and all the things we deal with at our work on digital collections, databases, APIs, tools and esoteric programming languages we prefer to write in. It is

 not

meant for a single subject, but covers all topics we 💖. So I hope you don’t mind if the next post is about one of my adorable cute rabbits.

Welcome to the Lab

Twitter for Goettingen Digitisation Centre / Göttinger Digitalisierungszentrum   Did you know, that with the current relaunch of the GDZ (Göttinger Digitalisierungszentrum), we implemented a small bridge to Twitter that tweets out the newly scanned works?  This feature was integrated in our Symfony App using the simple endroid/twitter library.  Feel free to follow this unofficial account at @just_scanned.

Twitter for Goettingen Digitisation Centre

Here is a cheat sheet to XQuery serialization.

XQuery Serialization Options

Looking for platforms to start prototyping the Atom text editor or its superior platform Electron passed our way at the Research and Development Department recently. For feasibility studies we sometimes test new frameworks and when we can use web technologies to start coding, there is quite a good chance for a seamless connection between data structures, algorithms and the front end.

Hacking the hackable editor

Docker activated “rate limits” by the end of 2020 - and as we are frequently pulling images from the main docker registry, we ran into trouble and our builds failed. But there is a quite simple solution for circumventing these rate limits - as we are using a gitlab runner on a dedicated server. This server already has the docker daemon installed and the solution is already documented.

Gitlab Runner and Docker's Rate Limits

Using a dependency proxy with npm, yarn or pnpm in CI or local development can greatly reduce outgoing traffic. Fortunately, FE now maintains an instance of such service. Verdaccio – not only a classical underpainting technique    The project maintains a current Helm chart that made its deployment into a Kubernetes cluster work like a charm.

Introducing a lightweight Node.js proxy registry service: Verdaccio

I tried to improve the downlad-time and size of the assets used in this blog. This is mainly about optimizing the images for web. Not only that all JavaScript from this site has been removed previously, the images were a huge factor in the traffic of this page..  The images used in this blog are of different types. We use PNG, JPEG and SVG. Some of these files are huge and not optimized for web use.

WebP

I wanted to switch a small website into a one-pager website lately, and as it so happend - the site was managed with TYPO3. Turns out, with TYPO3 it’s a piece of cake. I already had a sitepackage and a template, so the only thing was to glue the pages in a string, make them distingusable from each other, and adjust the menu. Taking care of the glue   The pages can be glued together with two viewhelpers from the vhs-extension.

Creating a onepager website with TYPO3

Web-scraping is ugly, but sometimes it may become necessary, because services don’t expose an API to retrieve data. Basically web-scraping is a mechanism to programatically open a website and grab the contents in order to using it for own purposes. The mission   In one of our sites, we are using the HTML output of a not-that-hip-anymore-database allegro.

Webscraping with Symfony

Central Logging Service for Distributed Applications   With the release of our great new presentation layer for the Goettingen Digitization Centre we faced new problems. The Application consists of many different separate services (and servers). For example a Symfony frontend application, separate applications to deliver and process IIIF manifests and images.

Logging for distributed applications

This article introduces an XQuery library that allows code generation according to the OpenAPI 3.0.2 standard based on RESTXQ. It uses xqDoc as well as XQuery function annotation with %rest and %test prefix, combined with type declaration and also searches for general information in package description resources (expath-pkg.xml and repo.xml). It ships with the “swagger-ui-dist” package grabbed from npm, but can be built without by ant.

OpenAPI Code Generator for RESTXQ

Heute soll es um die Werte in Scrum gehen. Ich glaube, dass gelebte Werte in der Teamarbeit eine absolute Grundvoraussetzung für ein High Performing Team sind. Zum Einstieg möchte ich mit ein paar kurzen einleitenden Worten zu den Scrum Werten beginnen. Wenn euch das neugierig macht, solltet ihr unbedingt den kurzen Methoden-Montag Podcast dazu anhören!

Werte bei Scrum

A data import is part of most of our projects and in some cases, we need to re-import all the data from time to time. This post shows how you can delegate this task to a GitLab CI job and start it by adding a certain keyword to your commit message. What is the problem? There are several circumstances when a data import (or a re-import) is required.

Data Import by CI Job

Symfony console   For a rather small task we needed to build a standalone command line app to process and copy some files. Our first approach to that included using the JVM (Java, Kotlin), because it needed to be deployable and runnable without any great hassle. But going a little further we skipped that approach and remembered the great Symfony Console which provides a quite simple interface.

Using Docker for a simple Symfony Console App

Dealing with Rancher Snapshots   A bare kubernetes cluster managed with rancher is hosted and provided by our computing center (GWDG). They offer great support when we have to scale the nodes, but after all some parts of the maintenance is our business. Setting the kubernetes version from a provided list or managing ingress controller or certbot are those tasks.

Resource Extraction from Rancher Snapshots

A centralized logging solution is essential for the efficient implementation of best practices for logging in Kubernetes. We present the FLOOD stack as an open source solution to this requirement. Logging in Kubernetes   The built-in logging in kubernetes only provides very basic functionality. Developers can only access the logs of one container at a time and there are only logs of the current pod and its direct predecessor available.

lab.sub - Articles