Getting Things Done in Genetics & Bioinformatics Research

Reposted from the original at https://blog.stephenturner.us/p/construct-objects-with-idiomatic-r-code

--- Today I discovered the constructive package and the construct() function for creating R objects with idiomatic R code to make human-readable reproducible examples.

 CRAN:

https://cran.r-project.org/package=constructive

 Source:

https://github.com/cynkra/constructive/

 Docs &amp;

Repost: Construct objects with idiomatic R code

I just got this announcement from EMBL-EBI about an RNA-seq/ChIP-seq analysis hands-on course. Find the full details, schedule, and speaker list here.

RNA-Seq &amp; ChiP-Seq Data Analysis Course at EBI

Stephen Turner

This blog has moved. This is reposted from Paired Ends:

 https://blog.stephenturner.us/p/planes-plausibility-analysis-of-epidemiological-signals-rplanes-r-package

PLANES provides a set of methods for evaluating the plausibility of epidemiological signals and forecasts.

PLANES: Plausibility Analysis of Epidemiological Signals

This is reposted from the original article:

Use R to prompt a local LLM with ollamar

A while back I wrote this post about how I stay current in bioinformatics &amp; genomics. That was

 nearly five years ago

. A lot has changed since then. A few links are dead.

Staying Current in Bioinformatics &amp; Genomics: 2017 Edition

Data “janitor-work”  The New York Times recently ran a piece on wrangling and cleaning data: “For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insights” Whether you call it “janitor-work,” wrangling/munging,  cleaning/cleansing/scrubbing, tidying, or something else, the article above is worth a read (even though it implicitly denigrates the important work that your housekeeping staff does). It’s one of the few “Big Data” pieces that

Do your "data janitor work" like a boss with dplyr

There were lots of interesting developments this month that didn't work their way into a full blog post. Here is an incomplete list of what I've been tweeting about over the last few weeks.

RNA-Seq Methods &amp; March Twitter Roundup

James Taylor came to UVA last week and gave an excellent talk on how Galaxy enables transparent and reproducible research in genomics. I'm gearing up to take on several projects that involve next-generation sequencing, and I'm considering installing my own Galaxy framework on a local cluster or on the cloud. If you've used Galaxy in the past you're probably aware that it allows you to share data, workflows, and histories with other users.

Guide to RNA-seq Analysis in Galaxy

You can find two  nice  recaps of last week's personalized  medicine policy forum on Genomics   Law Report and 23andMe's blog, The   Spittoon.

23andMe GCPM recaps and FDA meeting on Laboratory Developed Tests

For those of you not following GGD on Twitter you may not have seen this - California State Senator Alex Padilla and 23andMe are hosting a policy forum entitled "Genomics and the Consumer: The Present and Future of Personalized Medicine" today in San Francisco.

Genomics and the Consumer: The Present and Future of Personalized Medicine

A few weeks ago I covered an R package for efficient mixed model regression that is capable of simultaneously accounting for both population stratification and relatedness to compute unbiased estimates of standard errors and p-values for genetic association studies.

Mixed linear model approach adapted for genome-wide association studies

This looks like a must-read for anyone starting out in computational biology without extensive experience at the command line.  The 135-page document linked at the bottom of the Google Group page looks like an excellent primer with lots of examples that could probably be completed in a day or two, and provides a great start for working in a Linux/Unix environment and programming with Perl.

Unix and Perl for Biologists

A short announcement - my friend, colleague, running partner, and GGD contributor Will Bush is now an assistant professor in the Department of Biomedical Informatics, and investigator in the Center for Human Genetics Research here at Vanderbilt.Getting Genetics Done by Stephen Turner is licensed under a Creative Commons Attribution (CC BY) License.

Professor Bush

Department of Biostatistics Seminar/Workshop Series:

 Statistical Methods for DNA Resequencing Analysis in Disease-Gene Studies

Wenyi Wang, Ph.D., Faculty Candidate Stanford Genome Technology Center, UC Berkeley Statistics 2:00-3:00pm Monday, February 15, 2010 MRB III Room 1220 Intended Audience: Persons interested in applied statistics, statistical theory, epidemiology, health services research, clinical trials

Seminar: Statistical Methods for DNA Resequencing Analysis in Disease-Gene Studies

I wish I would have discovered this long ago.  Loading data into R or MySQL requires you to specify the full path to the file.  If you do this on a Windows machine there are two annoyances.  First, if you save something to your desktop the path to your desktop is really long.  Second, windows by default uses backslashes "\" in the file path, while R or other software requires forward slashes "/".

ClipPath copies filename and path from windows for loading into R

The Vanderbilt Kennedy Center is offering a free (repeat, free) one-day introductory course to the R statistical computing language on June 23, taught by Theresa Scott from the department of Biostatistics. You can find contact/registration info at the link below. Vanderbilt Kennedy Center - An Introduction to the Fundamentals &amp;

Free one-day R course at Vanderbilt

Has anyone ever used Galaxy? I saw their presentation at last year's ASHG. Seems like a great way to do collaborate on and keep a record of analyses in an easy web-GUI interface without having to download any software.

Has anyone ever used Galaxy?

AAAS and the Food and Drug Law Institute (FDLI) are holding three 2-day meetings in Washington DC on personalized medicine, addressing the scientific discoveries, business models, and policy changes that are necessary to develop personalized treatments and diagnostics. All three meetings will be available via a free live webcast where viewers can use response technology to interact with speakers and session chairs.

AAAS personalized medicine meeting webcast

Reposted from

 https://blog.stephenturner.us/p/tech-im-thankful-for-2024

Data science and bioinformatics tech I'm thankful for in 2024: tidyverse, RStudio, Positron, Bluesky, blogs, Quarto, bioRxiv, LLMs for code, Ollama, Seqera Containers, StackOverflow, ...It’s a short week here in the US. As I reflect on the tools that shape modern bioinformatics and data science it’s striking to see how far we’ve come in the 20 years I’ve

Tech I'm thankful for (repost)

I recently reformatted my laptop and needed to reinstall R and all the packages that I regularly use. In a previous post I covered R Commander, a nice GUI for R that includes a decent data editor and menus for graphics and basic statistical analysis.

Install and load R package "Rcmdr" to quickly install lots of other packages

A few folks have asked me how I get my news and stay on top of what's going on in my field, so I thought I'd share my strategy.

How to Stay Current in Bioinformatics/Genomics

Here are the papers we talked about in yesterday's Journal Club:  Genome Biol. 2009; 10(11): R134 Searching for SNPs with cloud computing. Langmead B, Schatz MC, Lin J, Pop M, Salzberg SL.  Table of Contents, Nature Methods, Visualization Supplement. Am J Hum Genet. 2010 Feb 12; 86(2): 113-25 Functional gene group analysis reveals a role of synaptic heterotrimeric G proteins in cognitive ability.

Papers from March 8, 2010 Journal Club

Last week I ran a one-day workshop on RNA-seq data analysis in the UVA Health Sciences Library. I set up an AWS public EC2 image with all the necessary software installed.

RNA-seq Data Analysis Course Materials

I direct the Bioinformatics Core at the University of Virginia, and I'm hiring. Visit this link on the UVA Jobs website for more information. Here's the description:  I'm Hiring - Bioinformatics Analyst in the UVA Bioinformatics CoreGetting Genetics Done by Stephen Turner is licensed under a Creative Commons Attribution (CC BY) License.

I'm Hiring!

Thomas Lasko, MD, PhD, from Google Inc., Mountain View CA, will be giving a seminar next week entitled:    "Spectral Anonymization Data"    March 3, 2010  12:00 - 1:00 p.m.  214 Light Hall    Abstract:     The great challenge of data anonymization is to condition a dataset for public release such that it remains scientifically useful, but it does not disclose sensitive information about the individuals it describes.

Seminar: Spectral Anonymization Data

A few weeks ago I suddenly reached the point that every graduate student once thought would never come - time to start writing my thesis. With a blank page and a blinking cursor staring me in the face it's time to compile all of my published and unpublished work I've accumulated over the last few years and wordsmith this pile of papers and results into a single cohesive unit.

Writing my Thesis - Follow me on Twitter

I just read a helpful paper on pathway analysis and interactome reconstruction: Tieri, P., Fuente, A. D., Termanini, A., &amp; Franceschi, C. (2011). Integrating Omics Data for Signaling Pathways, Interactome Reconstruction, and Functional Analysis. In

 Bioinformatics for Omics Data, Methods in Molecular Biology, vol.

Resources for Pathway Analysis, Functional Annotation, and Interactome Reconstruction with Omics Data

When I started grad school I started using Reference Manager (RefMan), similar to EndNote, to manage my references and bibliographies. It's a real pain, and I often feel like I'm powering my computer with the endless pumping and clicking of the mouse that it takes to import a reference into my library. Recently I've started using Zotero because of how easy it is to import references, store PDFs, and sync between computers.

Which Reference Management Software do you use? (Reader Poll)

Do you use SSH to connect to a remote Linux machine from your local Windows computer?  Ever needed to run a program on that Linux machine that displays graphical output, or uses a GUI? I was in this position last week trying to make figures using ggplot2 in R of results from an analysis of GWAS data which required using a 64-bit Linux machine with more RAM than my 32-bit windows machine can see.

Use PuTTY and XMing to see Linux graphics via SSH on your Windows computer

My new blog/newsletter ("Paired Ends") is now at blog.stephenturner.us. I'll be posting semi-regular updates and literature highlights in bioinformatics, computational biology, and data science, along with the occasional post on programming.

Moving to blog.stephenturner.us (Paired Ends)

The first ever RStudio conference was held January 11-14, 2017 in Orlando, FL. For anyone else like me who spends hours each working day staring into an RStudio session, the conference was truly

 excellent

. The speaker lineup was diverse and covered lots of areas related to development in R, including the tidyverse, the RStudio IDE, Shiny, htmlwidgets, and authoring with RMarkdown.

RStudio Conference 2017 Recap

This week, I'm off to Montreal for the International Congress on Human Genetics and I hope to see you there! If you are attending and are already a part of the Twitterverse, bring a tablet or phone and tweet away about the meeting using the official hashtag,  #ICHG2011.

Find me at ICHG!

Current Opinion in Microbiology

recently published a special issue in genomics. In an excellent editorial overview, “Genomics: The era of genomically-enabled microbiology”, Neil Hall and Jay Hinton give an overview of the state of the field in microbial genomics, summarize recent contributions, and give a great synopsis of each of the reviews in this issue.

Microbial Genomics: the State of the Art in 2015

I forgot where I originally found the code to do this, but I recently had to dig it out again to remind myself how to draw two different

 y

axes on the same plot to show the values of two different features of the data. This is somewhat distinct from the typical use case of aesthetic mappings in ggplot2 where I want to have different lines/points/colors/etc. for the

 same

feature across multiple subsets of data.

R: single plot with two different y-axes

Recently I tried compiling Eigensoft on my Ubuntu 10.10 Linux system running in Virtualbox and had no success. From comments on this blog post, it looks like the newer Ubuntu distros don't have the libg2c0 and related libraries (which were a part of the gcc3) and gcc4 uses gfortran instead.

Shellfish for Parallel PCA on GWAS data (Alternative to Eigenstrat)

This is the first in a series of posts on how to use MySQL with genetic data analysis. MySQL is a very popular, freely available database management system that is easily installed on desktop computers (or Linux servers).  The "SQL" in MySQL stands for Structured Query Language, which is by my humble estimation the most standardized way to store and access information in the known universe.

Using MySQL to Store and Organize Results from Genetic Analyses

Much of the work that bioinformaticians do is munging and wrangling around massive amounts of text. While there are some "standardized" file formats (FASTQ, SAM, VCF, etc.) and some tools for manipulating them (fastx toolkit, samtools, vcftools, etc.), there are still times where knowing a little bit of Unix/Linux is extremely helpful, namely awk, sed, cut, grep, GNU parallel, and others.

Useful Unix/Linux One-Liners for Bioinformatics

This builds on a previous post from Stephen. I was recently running a series of ANOVA analyses, and I used the aov() function because it had a few options that I preferred. Much like lm(), the function returns an object that you typically pass to summary() to view and interpret the output. It took me a bit of playing to figure out how to extract the information I needed.

Extracting values from R summary objects

Your Rprofile is a script that R executes every time you launch an R session.

Sync Your Rprofile Across Multiple R Installations

Lately I haven't written as many full length posts as usual, but here's a quick roundup of a few links I've shared on Twitter (@genetics_blog) over the last week:

First, 23andMe is having a big DNA Day Sale ($108) for the kit + 1 year of their personal genome subscription service https://www.23andme.com/. Previously mentioned R IDE RStudio released a new beta (version 0.93) that includes several new features and bugfixes.

Monday Links: 23andMe, RStudio, PacBio+Galaxy, Data Science One-Liners, Post-Linkage RFA, SSH

Last Year I introduced you to R Commander, a nice graphical user interface (GUI) for R for those of you who are still hesitant to leave the clicky-box style research a la SPSS for the far more superior reproducible research using R. As most of you know I'm a huge fan of ggplot2. Many of you came to the short course Hadley Wickham gave here a few weeks ago on ggplot2 and plyr.

Deducer: R and ggplot2 GUI

A tip of the hat to @JVJAI for pointing out this interesting looking paper in AJHG. ROADTRIPS: Case-Control Association Testing with Partially or Completely Unknown Population and Pedigree Structure Timothy Thornton and Mary Sara McPeek Department of Biostatistics, University of Washington.

 Abstract:



Genome-wide association studies are routinely conducted to identify genetic variants that influence complex disorders.

ROADTRIPS: Case-Control Association Testing with Partially or Completely Unknown Population and Pedigree Structure

"The hunt for the genetic roots of common diseases has hit a blank wall."

...quoting the first sentence in Nicholas Wade's New York Times article reviewing this PLoS Biology research paper by David Goldstein and his colleagues at Duke University. Also be sure to see Richard Robinson's synopsis of this paper, both published this week in PLoS Bio.

Rare Variants Create Synthetic Genome-Wide Associations

A while back, Stephen wrote a very nice post about converting PLINK output to a CSV file. If you are like me, you have used this a thousand times -- enough to get tired of typing lots of SED commands. I just crafted a little BASH script that accomplishes the same effect with a single easy to type command. Insert the following text into your .bashrc file.

Convert PLINK output to CSV Revisited

Hierarchical clustering is a technique for grouping samples/data points into categories and subcategories based on a similarity measure. Being the powerful statistical package it is, R has several routines for doing hierarchical clustering. The basic command for doing HC is        Nearly all clustering approaches use a concept of distance.

Hierarchical Clustering in R

Avi Ma'ayan, Ph.D., Assistant Professor in the Department of Pharmacology and Systems Therapeutics of Mount Sinai School of Medicine, will present "Network Analysis in Systems Biology", at 10:00 am on Wednesday, April 22 in Room 206 PRB.

Network analysis in systems biology workshop

Note 2010-11-17: there's more than one way to do this.

Embed R Code with Syntax Highlighting on your Blog

A live webcast of the House Committee on Energy and Commerce hearing on “Direct-to-Consumer Genetic Testing and the Consequences to the Public Health" is available at this link. I had trouble viewing the webcast in firefox - had to save the link and open it with VLC media player to get it working. You can also follow the #HouseDTC hastag on Twitter.

Webcast this Morning: House Committee on Energy and Commerce hearing on DTC Genetic Testing

Lucila Ohno-Machado, Professor of Medicine and Chief of the Division of Biomedical Informatics at UC-San Diego, will be giving a talk on "Accuracy of Individualized Risk Estimates for Personalized Medicine" next week, August 18, noon-1pm in 202 Light Hall. This should be an interesting perspective from a scientist with medical training on the utility of personal genomics tools in making healthcare decisions.

Accuracy of Individualized Risk Estimates for Personalized Medicine

Vanderbilt ACCRE users with PCs only...    If you use ACCRE to run multi-processor jobs you'll be glad to know that they now allow you to map your home directory to your local desktop using Samba (so you can access your files through My Computer as you normally would with local files).

Sync your home directories on ACCRE and the local Linux servers (a.k.a. "the cheeses")

A quick announcement about a formatting fix here - you can now print posts from GGD a little more cleanly.  When you print it should no longer include the title and sidebar, so most posts should now only use a page or two.Getting Genetics Done by Stephen Turner is licensed under a Creative Commons Attribution (CC BY) License.

GGD posts now printer-friendly

Logan recently emailed me an article in the New York Times about single-molecule DNA sequencing and I realized I knew next to nothing about the new and emerging technology that will change the way we do association studies (that is, if we're still even trying to find genetic associations in the first place).  The Wellcome Trust posted a news feature a few weeks back giving brief explanations and short videos on DNA sequencing, starting with the

Next-Gen Sequencing

Aviad Klein over at My ContRibution wrote a convenient R function to list the classes of all the vectors that make up a data.frame. You would think apply(kyphosis,2,class) would do the job but it doesn't - it calls every vector a character class. Aviad wrote an elegant little function that does the job perfectly without having to load any external package:

Quickly Find the Class of data.frame vectors in R

About a year ago I wrote a post about producing scatterplot matrices in R. These are handy for quickly getting a sense of the correlations that exist in your data. Recently someone asked me to pull out some relevant statistics (correlation coefficient and p-value) into tabular format to publish beside a scatterplot matrix.

More on Exploring Correlations in R

I just read an interesting paper on pathogen discovery using next-generation sequencing data, recommended to me by Nick Loman. A previously described algorithm (PathSeq, Kostic et al) for discovering microbes by deep-sequencing human tissue uses computational subtraction, whereby the initial collection of reads is depleted of human DNA by consecutive alignment to the human reference using MAQ and BLAST.

Identifying Pathogens in Sequencing Data

The Systems Biology Graphical Notation (SBGN) project is an effort to standardize the graphical notation used in diagrams of pathways, biochemical processes, and cellular processes studied in systems biology. SBGN defines a comprehensive set of symbols with precise semantics, together with detailed syntactic rules defining their use and how diagrams are to be interpreted.

Systems Biology Graphical Notation

Update 2010-04-21

: I forgot to post the link last time. That would have been helpful. Here you go: Vanderbilt Biostatistics: Statistical Problems to Document and to Avoid ..... At the Regression Modeling Strategies course I attended a few weeks ago, Frank Harrell pointed out the checklist on the biostatistics department's website of statistical problems to document and avoid.

Checklist: Statistical Problems to Document and Avoid

Check out this month's table of contents in  Nature Methods. It contains a series of five commissioned Reviews discuss the challenges of visualizing biological data and the visualization tools available to biologists working with genomes, alignments and phylogenies, macromolecular structures, images and systems biology data.

Nature Methods: Visualization

Writing from the previously mentioned intro to R course at the Kennedy Center. If you couldn't make it you can download all the course materials from Theresa Scott's website, under the "Current Teaching Material" heading. Here is a direct link to the PDF for the overview materials that we're going over today, along with the R code from the examples in this guide.

PDF tutorial from R course (Introduction to R)

Christen J. Willer, (of the Willer et al Nature Genetics lipid genetics paper), postdoc at University of Michigan and departmental Faculty Candidate, will be giving a talk tomorrow (Wednesday Jan 20) at noon in 208 Light Hall. I'll be going to this one. Topic:

Seminar: Cristen Willer - Mapping Genes for Complex Genetic Diseases: Lessons from Obesity and Lipid Levels

Head over to Revolutions Blog for a list of PDF and powerpoint resources for making the transition to R from other programming or stats languages.  All of these notes come from the New York R meetup. I enjoyed browsing the meetup's files - lots of powerpoints, PDFs, and example R data files for various topics, including several slideshows on ggplot2.

Coming to R from SQL, Python, SAS, Matlab, or Lisp

Strict quality control procedures are extremely important for any genome-wide association study.  One of the first steps you should take when running QC on your GWAS is to look for related samples in your dataset.  This does two things for you.

Visualizing sample relatedness in a GWAS using PLINK and R

Some of the best figure design ideas come from cartographers. If you've ever read a Tufte book you've seen lots of examples. Let's talk about using color effectively. Penn State geography professor Cindy Brewer's ColorBrewer tool for selecting color schemes for figures has been conveniently packaged into an R library called RColorBrewer.

Use meaningful color schemes in your figures

Here are citations for the articles discussed at our most recent meeting (June 24). Our next meeting is scheduled for July 17. ~Julia Choi Y, Wijsman EM, Weir BS. Case-control association testing in the presence of unknown relationships.

 Genet Epidemiol

. 2009 Mar 30. [Epub ahead of print] Gao X, Becker LC, Becker DM, Starmer JD, Province MA. Avoiding the high Bonferroni penalty in genome-wide association studies.

PCG Journal Club Articles, 6/24

A couple of weeks ago I, with the help of others here at UVA, organized a Software Carpentry bootcamp, instructed by Steve Crouch, Carlos Anderson, and Ben Morris. The day before the course started, Charlottesville was racked by nearly a foot of snow, widespread power outages, and many cancelled incoming flights. Luckily our instructors arrived just in time, and power was (mostly) restored shortly before the boot camp started.

Software Carpentry Bootcamp at University of Virginia

Metagenomics

is the study of DNA collected from environmental samples (e.g., seawater, soil, acid mine drainage, the human gut, sputum, pus, etc.). While traditional microbial genomics typically means sequencing a pure cultured isolate, metagenomics involves taking a culture-free environmental sample and sequencing a single gene (e.g. the 16S rRNA gene), multiple marker genes, or shotgun sequencing everything in the sample in order to

Comparing Sequence Classification Algorithms for Metagenomics

I use this all the time, and the setup is dead simple. Follow the code below to load the RMySQL package, connect to a database (here the UCSC genome browser's public MySQL instance), set up a function to make querying easier, and query the database to return results as a data frame.    Getting Genetics Done by Stephen Turner is licensed under a Creative Commons Attribution (CC BY) License.

Query a MySQL Database from R using RMySQL

PLINK/SEQ is an open source C/C++ library for analyzing large-scale genome sequencing data. The library can be accessed via the pseq command line tool, or through an R interface. The project is developed independently of PLINK but it's syntax will be familiar to PLINK users. PLINK/SEQ boasts an impressive feature set for a project still in the beta testing phase.

PLINK/SEQ for Analyzing Large-Scale Genome Sequencing Data

Just caught this on the OpenHelix Blog. In an interview with Charlie Rose, NIH director Francis Collins said Computational biologists will be the "breakthrough" artists of the future.    Getting Genetics Done by Stephen Turner is licensed under a Creative Commons Attribution (CC BY) License.

Francis Collins: Computational biologists are "breakthrough artists"

Background  This tutorial shows an example of RNA-seq data analysis with DESeq2, followed by KEGG pathway analysis using GAGE. Using data from GSE37704, with processed data available on Figshare DOI: 10.6084/m9.figshare.1601975.

Tutorial: RNA-seq differential expression &amp; pathway analysis with Sailfish, DESeq2, GAGE, and Pathview

I first mentioned Coursera about a year ago, when I hired a new analyst in my core. This new hire came in as a very competent Python programmer with a molecular biology and microbial ecology background, but with very little experience in statistics.

Coursera Specializations: Data Science, Systems Biology, Python Programming

Nice review on the genetics of quantitative traits with good figures and references published this morning in Nature Reviews Genetics.   Abstract:  The genetics of quantitative traits: challenges and prospects (NRG AOP)Getting Genetics Done by Stephen Turner is licensed under a Creative Commons Attribution (CC BY) License.

The genetics of quantitative traits: challenges and prospects

Reposted from https://blog.stephenturner.us/p/duckdb-vs-dplyr-vs-base-r.


 
 TL;DR
 
 : For a very simple analysis (means by group on 100M rows), duckdb was 125x faster than base R, and 28x faster than readr+dplyr, without having to read data from disk into memory. The duckplyr package wraps DuckDB's analytical query processing techniques in a dplyr-compatible API.

DuckDB vs dplyr vs base R

A couple of months ago I posted about how to visualize exome coverage with bedtools and R. But if you're looking to get a basic handle on genome arithmetic, take a look at Aaron Quinlan's bedtools tutorials from the 2013 CSHL course.

Bedtools tutorial from 2013 CSHL course

Reposted from the original at https://blog.stephenturner.us/p/github-repo-to-text-for-llm-input.

---  If you use ChatGPT, Claude, or even some local model through Ollama or HuggingFace Assistants, you’ll know that the chat interface makes it challenging to feed in an entire repo like a Python or R package, because functions, tests, etc. can be scattered across many files throughout a repo.

Turn a GitHub repo into a single text file for LLM-friendly input (repost)

Reposted from

 Paired Ends

at https://blog.stephenturner.us/p/llm-translate-documentation. ---The lang package overrides the ? and help() functions in your R session. The translated help page will appear in the help pane in RStudio or Positron. It can also translate your Roxygen documentation.

Use an LLM to translate help documentation on-the-fly

This is reposted from the original at https://blog.stephenturner.us/p/use-nanoparquet-instead-of-readr-csv.Parquet is interoperable between Python and R, fast to read+write, works well with databases, and stores complex data types (e.g., tibble listcols). Use it instead of CSV. Many pros, few (no?) cons. Yesterday I wrote about base R vs. dplyr vs. duckdb for a simple summary analysis.

Use nanoparquet instead of readr/CSV

This blog has moved. This is reposted from Paired Ends:


 https://blog.stephenturner.us/p/create-a-free-llama-405b-llm-chatbot-github-repo-huggingface

Llama 3.1 405B is the first open-source LLM on par with frontier models GPT-4o and Claude 3.5 Sonnet.

Create a free Llama 3.1 405B-powered chatbot on any GitHub repo in 1 minute (cross-posted from Paired Ends)

This is re-posted from my newsletter, where I'll be posting from now on: https://blog.stephenturner.us/p/biorecap-r-package-for-summarizing-biorxiv-preprints-local-llm    ---TL;DR I wrote an R package that summarizes recent bioRxiv preprints using a locally running LLM via Ollama+ollamar, and produces a summary HTML report from a parameterized RMarkdown template.

biorecap: an R package for summarizing bioRxiv preprints with a local LLM

How many reads do I need? What's my sequencing depth?

These are common questions I get all the time. Calculating how much sequence data you need to hit a target depth of coverage, or the inverse, what's the coverage depth given a set amount of sequencing, are both easy to answer with some basic algebra. Given one or the other, plus the genome size and read length/configuration, you can calculate either.

Covcalc: Shiny App for Calculating Coverage Depth or Read Counts for Sequencing Experiments

A while back I showed you how to make volcano plots in base R for visualizing gene expression results. This is just one of many genome-scale plots where you might want to show all individual results but highlight or call out important results by labeling them, for example, with a gene name. But if you want to annotate lots of points, the annotations usually get so crowded that they overlap one another and become illegible.

Repel overlapping text labels in ggplot2

I work with

 gene lists

on a nearly daily basis. Lists of genes near ChIP-seq peaks, lists of genes closest to a GWAS hit, lists of differentially expressed genes or transcripts from an RNA-seq experiment, lists of genes involved in certain pathways, etc. And lots of times I’ll need to convert these gene IDs from one identifier to another. There’s no shortage of tools to do this. I use Ensembl Biomart.

Annotables: R data package for annotating/converting Gene IDs

Joanna Zhao’s and Jenny Bryan’s R graph catalog is meant to be a complement to the physical book, Creating More Effective Graphs, but it’s a really nice gallery in its own right. The catalog shows a series of different data visualizations, all made with R and ggplot2. Click on any of the plots and you get the R code necessary to generate the data and produce the plot.

R + ggplot2 Graph Catalog

Anscombe’s quartet comprises four datasets that have nearly identical simple statistical properties, yet appear very different when graphed. Each dataset consists of eleven (

 x

,

 y

) points. They were constructed in 1973 by the statistician Francis Anscombe to demonstrate both the importance of graphing data before analyzing it and the effect of outliers on statistical properties. Let’s load and view the data.

Using and Abusing Data Visualization: Anscombe's Quartet and Cheating Bonferroni

I have a noteworthy blogs tag on this blog that I sort of forgot about, and haven't used in years. But I started reading one recently that's definitely qualified for the distinction. The Microbiome Digest is written by Elisabeth Bik, a scientist studying the microbiome at Stanford.

Microbiome Digest Blog

Last week I taught a three-hour introduction to R workshop for life scientists at UVA's Health Sciences Library. I broke the workshop into three sections: In the first half hour or so I presented slides giving an overview of R and why R is so awesome. During this session I emphasized reproducible research and gave a demonstration of using knitr + rmarkdown in RStudio to produce a PDF that can easily be recompiled when data updates.

Introduction to R for Life Scientists: Course Materials

I've been asked a few times how to make a so-called volcano plot from gene expression results. A volcano plot typically plots some measure of effect on the x-axis (typically the fold change) and the statistical significance on the y-axis (typically the -log10 of the p-value). Genes that are highly dysregulated are farther to the left and right sides, while highly significant changes appear higher on the plot.

Using Volcano Plots in R to Visualize Microarray and RNA-seq Results

I recently found this little gem of a web app that analyzes the clarity of your writing. Hemingway highlights long, complex, and hard to read sentences. It also highlights complex words where a simple one would do, and highlights adverbs, suggesting you use a stronger verb instead. It highlights passive voice (bad!), and tells you the minimum reading grade level necessary to understand your writing.

Unsuck your writing

Since the near beginning of genome-wide association studies, the PLINK software package (developed by Shaun Purcell’s group at the Broad Institute and MGH) has been the standard for manipulating the large-scale data produced by these studies.

PLATO, an Alternative to PLINK

Several post-publication peer review forums already exist, such as Faculty of 1000 or PubPeer, that facilitate discussion of papers after they have already been published. F1000 only allows a small number of "faculty" to comment on articles, and access to read commentary requires a paid subscription. PubPeer and similar startup services lack a critical mass of participants to make such a community truly useful.

PubMed Commons: One post-publication peer review forum to rule them all?

It's happened to all of us. You read about a new tool, database, webservice, software, or some interesting and useful data, but when you browse to http://instititution.edu/~home/professorX/lab/data, there's no trace of what you were looking for.

 THE PROBLEM

This isn't an uncommon problem. See the following two articles: The first gives us some alarming statistics.

Stop Hosting Data and Code on your Lab Website

Kevin White from the University of Chicago will be giving a special guest lecture at NCI next week on systems biology approaches to mine genomics data for biomarkers and therapeutic targets. The lecture will be available online as a videocast.

Webinar: Genomic Networks - Resolving Biomarkers from a Cloud of Data

I just got an email from Illumina about a webinar that looks interesting this Wednesday at 9am PST (noon EST) on clinical applications of next-gen sequencing. Date: Wednesday, December 7, 2011 Time: 9:00 AM (PST) Speaker: Rick Dewey, MD, Stanford Center for Inherited Cardiovascular Disease        Next-generation sequencing (NGS) presents both challenges and opportunities for clinical care.

Webinar: Applications of Next-Generation Sequencing in Clinical Care

The Galaxy Project started using CiteULike to organize papers that are about, use, or reference Galaxy. The Galaxy CiteULike group is open to any CUL user, and once you join, you can add papers to the group, assign tags, and rate papers.

Galaxy Project Group on CiteULike and Mendeley

After evaluating an unnamed bioinformatics core facility, a group of bioinformaticians in Europe wrote up a short list of basic guidelines for organizing a bioinformatics core facility in large research institutes.

How to Run a Bioinformatics Core

I just accepted an offer for a faculty position at the University of Virginia in the Center for Public Health Genomics / Department of Public Health Sciences. Starting in October I will be developing and directing a new centralized bioinformatics core in the UVA School of Medicine. Over the next few weeks I'm taking a much-needed vacation next door in Kauai and then packing up for the move to Charlottesville.

I'm Starting a New Position at the University of Virginia

Sequencing company Complete Genomics recently made available 69 ethnically diverse complete human genome sequences: a Yoruba trio; a Puerto Rican trio; a 17-member, 3-generation pedigree; and a diversity panel representing 9 different populations. Some of the samples partially overlap with HapMap and the 1000 Genomes Project. The data can be downloaded directly from the FTP site.

Download 69 Complete Human Genomes

There will be a (free) tutorial on the 1000 genomes project at this year's ASHG meeting on Wednesday, November 3, 7:00 – 9:30pm. You can register online at the link below.

Tutorial on the 1000 Genomes Project Data

A District Court handed down a summary judgment invalidating most of Myriad's claims to both the BRCA1 DNA sequence and the method of testing for early-onset familial breast and ovarian cancer. See Genetic Future and Genomics Law Report for analysis.Getting Genetics Done by Stephen Turner is licensed under a Creative Commons Attribution (CC BY) License.

Federal Courts Invalidate Myriad's Breast Cancer Gene Patents

Vanderbilt Center for Human Genetics Research faculty candidate Goncalo Abecasis will be interviewing for a faculty position here this week. Come check out his seminar - "Sequencing Thousands of Human Genomes" - Friday June 11th, 2-3pm in 512 Light Hall.Getting Genetics Done by Stephen Turner is licensed under a Creative Commons Attribution (CC BY) License.

Getting Genetics Done

Start using RSS in 5 minutes

GGD Launches!