Martin Modrák

In this post we’ll explore a particular link between Bayes factors and cross-validation I was introduced to via Fong &amp; Holmes 2020. I’ll then argue why this is a reason to not trust Bayes factors too much. This is a followup to Three ways to compute a Bayes factor, though I will repeat all the important bits here.

Cross-validation — a fourth way to compute a Bayes factor

The Approximation - Big Picture Saddlepoint for Sum of NBs Implementing the Approximation in Stan A Simple Baseline Eyeballing Masses Evaluating Performance Summing up Saddlepoint Approximations for Other Families  I recently needed to find the distribution of sum of non-identical but independent negative binomial (NB) random variables. Although for some special cases the sum is itself NB, analytical solution is not feasible in the general case.

Approximate Densities for Sums of Variables: Negative Binomials and Saddlepoint

brms is a great package. It allows you to put predictors on a lot of things. Its power is however not absolute — one thing it doesn’t let you directly do is use data to predict variances of random/varying effects.

Brms hacking: linear predictors for random effect standard deviations

To celebrate a new paper out in Bayesian Analysis, let’s talk simulation-based calibration checking (SBC). SBC is a method where you use simulated datasets to verify that you implemented you model correctly and/or that your sampling algorithm work. It  was introduced by Talts et al. and has been known and used for a while, but was considered to have a few shortcomings, which we try to address.

The SBC package - check your models before you wreck yourself

Recently another high-profile piece on abandoning statistical significance by Amrhein, Greenland &amp; McShane was published. I have mixed feelings about this, me and my Twitter bubble are mostly like “Another one of those?!”… But how did I get from not doing almost any statistics five years ago to considering myself a cool insider that can look down on a prominent piece by a group of lifelong experts?

Thank you: Statistics as a Journey

For the Czech national bioinformatics conference (ENBIK) I prepared a short presentation on Type S and Type M errors and how to use simulations to understand what your method might do before conducting an experiment. I show how t-test can fail, inspired by Andrew Gelman’s take on power = .06 and how DESeq2 (used to determine differentially expressed genes) does a good job at mitigating false positives at the cost of increased false negatives.

Kangaroo and DESeq2 (ENBIK 2018)

This is a second post in my series on taming divergences in Stan models, see the first post in the series for a general introduction. Also see guide to Stan warnings


 Standard caveat:


 I am not an expert on Stan, I consider myself just an advanced user who likes to explain things.

Identifying non-identifiability

I have just finished my submission to StanCon Helsinki 2018 which deals with my ongoing work on Bayesian modelling for gene regulation. I submitted late, because I procrastinated a lot on it and I recently have terrible time management in general. So we will se if it will even get considered.  View the compiled submission or check the Genexpi-Stan GitHub repository, to view the source code.

ODE Model of Gene Regulation (for StanCon)

This is a neat trick I found on Tyler Morgan-Wall’s Twitter and is originally attributed to Joe Cheng. You can run any Shiny app without blocking the session. My helper function to run ShinyStan without blocking is below: launch_shinystan_nonblocking &lt;- function(fit) {   library(future)   plan(multisession)   future(     launch_shinystan(fit) #You can replace this with any other Shiny app   ) }  Hope that helps!

Launch Shiny App Without Blocking the Session

This is a personal webpage/blog of Martin Modrák, an academic researcher in bioinformatics, currently at Institute of Microbiology of the Czech Academy of Sciences, laboratory of bioinformatics. I also do a bit of statistical consulting on other people’s projects. Get in touch! I also used to do research in AI for computer games. And I have a defunct software engineering blog.

About

Although my time with the Stan language has been enjoyable, there is one thing that is not fun when modelling with Stan. And it is the dreaded warning message: There were X divergent transitions after warmup. Increasing adapt_delta above 0.8 may help.

Taming Divergences in Stan Models

I’ve read Yihui Xie’s thoughtful response to the I don’t like notebooks talk from JupyterCon 2018. And I agree with basically everything Yihui said, only one point felt like it could give a wrong impression. It states:  This reads as if there is no room for automated tests in markdown/notebooks.

A Plea for Tests in R Markdown

Nathaniel Haines made a neat tweet showing off his model of reaction times that handles possible contamination with both implausibly short reaction times (e.g., if people make an anticipatory response that is not actually based on processing the stimulus of interest) or implausibly large reaction times (e.g., if their attention drifts away from the task, but they snap back to it after having “zoned out” for a few seconds). Response times that

Using brms to model reaction times contaminated with errors

So I recently started programming in Elm and also did some stuff in Rust. Honestly, it was mostly a hype-driven decision, but the journey was definitely worth it. I also noticed that although those two languages differ in their target audience and use cases, they made many similar design decisions. I think this is no coincidence.

What Elm and Rust Teach us About the Future

Not long ago, I came across a nice blogpost by Kahtryn Morrison called A gentle INLA tutorial. The blog was nice and helped me better appreciate INLA. But as a fan of the Stan probabilistic language, I felt that comparing INLA to JAGS is not really that relevant, as Stan should - at least in theory - be way faster and better than JAGS.

A Gentle Stan vs. INLA Comparison

I am not a staunch advocate of Bayesian methods — I can totally see how for some questions a frequentist approach may provide more satisfactory answers. In this post, we’ll explore how for a simple scenario (negative binomial regression with small sample size), standard frequentist methods fail at being frequentist while standard Bayesian methods provide good frequentist guarantees.

Using Bayesian tools to be a better frequentist

Generating document via RMarkdown is fun! So I recently used RMarkdown to generate reports that were written in Czech. Interestingly, Czech has rules on some words that are not allowed to be the last on a line of text - those are almost all single-letter words and a few abbreviations. MS Word is actually smart enough to enforce this policy, but this does not happen for the HTML and PDF outputs from RMarkdown.

Enforcing line-break rules in RMarkdown via Pandoc

Sometimes you are developing a model that has multiple variants: maybe you want to consider several different link functions somewhere deep in your model, or you want to switch between estimating a quantity and getting it as data or something completely different. In these cases, you might have wanted to use optional parameters and/or data that apply only to some variants of your model.

Optional Parameters/Data in Stan

I love thinking about other people’s statistical problems! So get in touch if you need help with: Developing and debugging statistical models (particularly Stan and INLA) Exploratory data analysis Statistical graphics Small Inquiries   If I have time, I will answer to small inquiries on Stan Discourse, tag me in as @martinmodrak to get my attention. You can check out list of my previous answers.

Statistical Consulting

This post was inspired by a very interesting paper on Bayes factors: Workflow Techniques for the Robust Use of Bayes Factors by Schad, Nicenboim, Bürkner, Betancourt and Vasishth. I would specifically recommend it for its introduction into what actually is a hypothesis in the Bayesian context and insights into what Bayes factors are.