If you wish to access the data from your MS-DIAL analysis programmatically for post-processing, then you face the non-trivial challenge of parsing the output to a dataframe. This is made challenging because the format of the alignment exports is not a simple xy matrix, but contains two data frames nested within each other, one for the qualitative information and the other for the quant. Consequently the headers for the qualitative data frame begin on row 5, as can be seen in this screenshot.

The qualitative data consists of 32 columns, beginning with Alignment ID, Average Rt(min), Average Mz, and Metabolite name and finishing with MS/MS spectrum. That last column is the point where things get interesting because the four rows above it in that 32nd column contain the headers for a separate table, aligned horizontally instead of vertically.

I have shaded and outlined the different columns for clarity. The blue table has headers in the left hand column (AF). The data includes class, file type, injection order and batch. This is essential data for creating quantitative plots, doing manual post-processing, such as blank subtraction or normalisation, and for many other downstream operations. You could enter this data into your processing as a separate input, but it is desirable to be able to scrape this data from a single input.

I have written a short python script in Jupyter Notebook to separate the qualitative data and the quantitative data from this single file and to scrape this important metadata and provide access to it as list objects. Please try it out and let me know if it useful.

I use our QExactive for untargeted metabolomics in the range m/z 80~1500. Typically with resolution set to the minimum value of 17,500, because the ability to scan quickly is more important than the ability to resolve m/z. The resolution on the Agilent QTOF in the Cancer Research lab is comparable, at ~20,000. I only mention this because it is another “high resolution MS” platform, as I understand the term. I consider these instruments worthy of the “high resolution” qualifier in HRMS because the other instruments I use are all “unit” mass spectrometers that differ in resolution to the QExactive and QTOF, as typically used, by factors in the tens of thousands.

However, as I’m sure you know, the QExactive can go up to resolutions of 140,000. Well, ours can. Newer models may be even more awesome. But the scan speed scales proportionally to the resolution. Consequently, I can’t use that high resolution when I analyse complex biological samples due to the large number of features in the data, the width of the peaks, the need to maintain throughput, & the need to collect multiple MS/MS scans in between full scans.

I recently tweeted a question asking whether the term “ultrahigh resolution mass spectrometers” was useful, when applied to the QExactive, in reference to a paper in ES&T: An Automated Methodology for Non-targeted Compositional Analysis of Small Molecules in High Complexity Environmental Matrices Using Coupled Ultra Performance Liquid Chromatography Orbitrap Mass Spectrometry. I can’t find the resolution used in the LC-MS method in the paper. However, if I assume it was the maximum resolution available on our own instrument that is still only 8 times higher than the value I routinely use and refer to as “high resolution” mass spec.

The seven Tesla Bruker Solarix at the UoA School of Biological Sciences apparently has a maximum resolution of 10 million. That is 71 times higher than the maximum on my QExactive and 571 times higher than the resolution I typically use it at. No one here typically refers to that as “ultrahigh” resolution mass spectrometry. It’s just mass spec. I’m not even sure what you’d use that resolution for, or whether it’s achievable in routine analysis. I think it’s something that protein chemists use to resolve things like post-translational modifications.

The division of analytical techniques into ‘good’, ‘better’ and ‘best’ evolves every day. As an example, a few years ago I was lucky enough to attend a seminar by Peter Derrick where he presented the results of the multi-turn TOF mass spec he had been working on for several years. When the instrument was first reported in 2005 the authors reported resolutions of 350,000 and termed this “very high resolution mass spec”. That resolution is now achieved every day by commercial Orbitrap instruments.

The prefix ‘ultra-‘ indicates extreme or transcendental status of the suffix. If the division between unit resolution and HRMS is in the tens of thousands, I do not think an increase of eight times above that worthy of that prefix. In fact, I reject the need to brand every modest increment in technical capability with an inappropriate adjective. Science evolves and technology that was once cutting edge becomes pedestrian and then obsolete. Science is also beset with jargon. The continuous addition of superlatives to describe the incremental progress in technology is unhelpful to students, who get lost in the (often abused) terminology. It is an impediment to efficient communication of methods between scientists and even more so to the lay public.

Don’t get me started on LC/HPLC/UPLC/UHPLC.

In my previous post I described how to convert NIST LC-MS/MS libraries to .msp format using the lib2nist program. This procedure will not work for the mainlib and replib EI libraries. For some reason these are encoded differently and will not appear in lib2nist as source libraries for conversion. The only way I have found to convert these is to use the Agilent version of the NIST library, which comes with the library in two different formats: the NIST format, which you can’t convert, and a .L format file used by Agilent’s MassHunter software suite. This latter file appears to contain all of the spectra from mainlib and replib, as well as some Kovats Retention Index data (of which more later).

To convert this .L format library using lib2nist you just have to select the file as the conversion source.

Once you have selected the source file to export from you need to select a subset fo the data to export as lib2nist will not export the whole thing in one go. I can’t remember how many files you should select but I think it has to be blocks of about 50K. There are nearly 400K spectra in NIST2020 so that’s eight subfiles you need to export and then combine.

Once you have exported the separate files you will need to use a large text file editor, such as Vim, to combine them into one file. Then you should be able to select that file in MS-DIAL to get identities for all of your features.

Thanks to Raquel Cumeras (Twitter @rcumeras) for reminding me how to do this.

Addition:

I have confirmed that lib2nist cannot export more than 64K spectra at a time when working on EI libraries. I tried exporting 250K, which works when you export the LC-MS/MS libraries. Here’s the error it threw:

NIST format libraries contain a lot of valuable information that would be even more useful if it could be accessed by other mass spectrometry software, such as MS-DIAL and AMDIS. I have been told that the NIST MS Search Program has an API which allows you to pass spectra to it and obtain results programmatically. This is evident in proprietary software, such as Agilent MassHunter and Thermo Freestyle. I understand you can email the NIST people and they will send you details of the API. However, open source software, such as MS-DIAL don’t seem to have implemented that capability. Therefore it is necessary to use other solutions to convert the information encoded in the NIST library into a format accessible by other software.

Public mass spectral libraries are typically encoded in several open formats, such as .mgf, .msp and .mzML formats. These formats are typically encoded as text and so are readily accessible. MS-DIAL uses .msp format libraries, for example. A description of the .msp text format can be found in this pdf (page 47). The lib2nist program included with the NIST library will allow you to convert the NIST library and similarly encoded data, such as the Wiley Registry, into these accessible formats. I have done this with both EI and HRMSMS libraries from NIST. Using lib2nist you can export each library to a series of .msp text files. You can’t export the whole library in one go so you have to limit the export to a certain number of records at a time. I found that 250K works. Once you’ve exported all your spectra from NIST you can recombine the files using a large text file editor, such as Vim. If you export the NIST high resolution MS/MS library the .msp file is nearly 3GB so there’s quite a bit of text.

The MS-DIAL team provide their own spectral libraries in .msp format incorporating all the available public libraries, such as those from mass spectral repositories, MassBank and GNPS, as well as several libraries kindly shared by individual research groups. If you open these .msp in a text editor and inspect the content you can see that the keys for the fields in the MS-DIAL libraries vary from those exported by lib2nist. The table below shows how they differ.

NIST	MS-DIAL	notes
Name	NAME
Notes
Precursor_type	PRECURSORTYPE	adduct
Spectrum_type		MS1/MS2
PrecursorMZ	PRECURSORMZ
Instrument_type
Instrument
Sample_inlet
Ionization
In-source_voltage
Collision_gas
Collision_energy	COLLISIONENERGY
Ion_mode	IONMODE	polarity
InChIKey	INCHIKEY
Synon
Formula	FORMULA
MW
ExactMass
CASNO
NISTNO
ID
	SMILES
	CCS
	Ontology	molecule class
	RETENTIONTIME
Comment	Comment
Num peaks	Num Peaks

Firstly, there’s a lot more fields in the NIST library export than the MS-DIAL library .msp file. Secondly, the fields are in lower case, with capitalised first letters and words separated by underscores. The exceptions are PrecursorMZ and Num peaks, neither of which get an underscore. There’s no obvious reason why.

The MS-DIAL fields aren’t consistent either, with a mixture of capitalised field names without spaces or underscores separating words, and words in lower case, with Num peaks matching the NIST format. The MS-DIAL documentation claims that case of the field text doesn’t matter. I’m guessing the underscores do as when I try importing the .msp exported by lib2nist I get no annotations in MS-DIAL. I wrote a Python script to substitute the MS-DIAL format keys for those in the lib2nist export. However, this didn’t work either. I then stripped out all of the fields from the NIST export that weren’t present in the MS-DIAL library. This did work and I got annotations from MS-DIAL!

For the final icing on this cake you can combine the MS-DIAL .msp library with your NIST export to get annotations from both.

NB

This procedure will not work for the NIST mainlib and replib EI libraries. See the follow-up post for instructions on how to convert them.

I want to illustrate LC-MS data in 3D to illustrate patterns of isotopes, adducts and molecular structures. I want my students to be able to present these as figure in their dissertations and theses and ultimately manuscripts. I’ve managed this using the Plotly library for Python. Here’s an example from my MSc students’ LC-MS analysis of sterol glucosides in lecithin:

The three ions at the top are acetic adducts of the three common plant sterol glucosides, whereas the three at the bottom are chlorine adducts of the same molecules. You can see they coelute and their m/z delta is 24. This is just an image, I haven’t worked out how to post interactive Plotly figures yet.

Here’s the code:

# This python script strips the time, m/z and intensity values from an LC-MS.txt file
# and plots it as a heatmap using the plotly library

# Copyright (C) 2018 Chris Pook

#This program is free software: you can redistribute it and/or modify
#it under the terms of the GNU General Public License as published by
#the Free Software Foundation, either version 3 of the License, or
#(at your option) any later version.

#This program is distributed in the hope that it will be useful,
#but WITHOUT ANY WARRANTY; without even the implied warranty of
#MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
#GNU General Public License for more details.

#You should have received a copy of the GNU General Public License
#along with this program. If not, see <http://www.gnu.org/licenses/&gt;.

 

# Start from a full scan (MS2 scan) mass spectrometry file in your native format
# convert this to an MS.txt file using msconvert (select text output)
# You can get msconvert from the Proteowizard package

# DON’T FORGET TO SET THE SCALING FACTOR
# numbers <1 flatten the data (less contrast)
# numbers >1 increase contrast
factor = 1

import glob, os
from Tkinter import Tk
from tkFileDialog import askopenfilename
import pandas as pd
import plotly.plotly as py
import plotly.graph_objs as go
import plotly.offline as offline

# get a MS txt file
Tk().withdraw() # we don’t want full GUI, so keep the root window from appearing
f = askopenfilename(filetypes=[(‘MS txt file’, ‘.txt’)],
title=’select one MS txt file’) # show an “Open” dialog box and return the path to the selected file
Folder = os.path.dirname(f) # strip out the folder path
aa = f.split(‘.’)
output = aa[0] + “_STACKED.csv”

# open the file and read it line-by-line
fOpen = open(f, “r”)
lines = fOpen.readlines()

# some variables
# the three arrays will end up holding lists of those values
# each triplet of values codes for one point on the heatmap
a = 0
secs = []
mz = []
signal = []
s = 0
m = 0
sig = 0

# now, let’s hack that data out!
for n in lines:
if ” cvParam: scan start time, ” in n:
# split off the text at the start of the row
b = n.split(‘,’)
#strip out whitespace
c = b[1].strip()
# convert to a float
s = float(c)
#print s
if ” binary: [” in n:
# split off the text at the start of the row
d = n.split(‘]’)
# strip the whitespace around the data
e = d[1].strip()
#turn it into a list of floats
f = [float(i) for i in e.split()]
#print f[0]
if a < 1:
m = f
a = 1
#print m[0]
#print m[1]
else:
#print f
sig = f
#print sig[0]
#print sig[1]
for g in range(0, len(m)-1):
#print g
#print s
secs.append(s)
#print len(secs)
#print m[0]
mz.append(m[g])
#print sig[0]
signal.append(sig[g])
a = 0

# a little debugging output
print len(secs)
print len(mz)
print len(signal)

# stick all that lovely data into a pandas dataframe
MS = [(‘seconds’,secs),
(‘m/z’, mz),
(‘signal’,signal),
]
df = pd.DataFrame.from_items(MS)

# more debugging output
print df.head()

# layout details you can change
# either uncomment autosize or specify your own height & width
layout = go.Layout(
width = 800,
height = 600,
#autosize = True,
#showlegend = False,
xaxis=dict(
title=’seconds’),
yaxis=dict(
title=’m/z’)
)

# it’s heatmap time!
# plotly colourscales – take your pick!
#Blackbody,Bluered,Blues,Earth,Electric,Greens,Greys,Hot,Jet,Picnic
#Portland,Rainbow,RdBu,Reds,Viridis,YlGnBu,YlOrRd

trace = go.Heatmap(x = df[‘seconds’], y = df[‘m/z’], z = pow(df[‘signal’], factor), colorscale= “Hot”)
data=[trace]

fig = go.Figure(data=data, layout=layout)
offline.plot(fig, filename=’my_awesome_LC-MS_heatmap.html’)

 

One of the cool sampling technologies we have here at AUT is Stir Bar Sorptive Extraction [SBSE]. The stir bars are little magnets coated with a layer of PDMS. You pop them into a liquid sample sat on a magnetic stirrer and over a period of a few minutes to an hour or more they will absorb any interesting organic compounds, even ones present at trace amounts, extracting and concentrating them from solution so that they can be analysed by GC-MS. In order to get your compounds off the stir bar and into the GC-MS you have to heat them to 300°C under an inert gas. Our Gerstel multipurpose sampler has a purpose-built doohicky to do this called a Thermal Desorption Unit. The heat vapourises the organic compounds from the PDMS trap and a flow of helium carries the organic compounds onto the GC column.

This is all very cool but before you can use the stir bars you need to thermally condition them to clean them of contaminants. The stir bars pick these contaminants up from anything they come into contact with. Its what they’re designed to do. However, it’s essential to be able to distinguish between contaminants the stir bars have picked up by accident and the interesting compounds I’d like to analyse. The thermal conditioning process releases any contaminants that might confound your analysis before you put the stir bars into your samples so it is an essential precursor to sensitive analysis. The problem is it takes up instrument time on the GC-MS. A lot of instrument time. The manufacturer recommends (pdf) that you condition stir bars for two hours prior to use. Analysing a single stir bar takes about an hour so total instrument time for one analysis is three hours.

I wanted an alternative way of thermally conditioning the stir bars that wouldn’t take up valuable instrument time. This would require some sort of enclosure that can be heated to 300°C and filled with a flow of inert gas. It has to be a flow to remove volatile contaminats as they evaporate. I have several other gas chromatographs in my lab besides the GC-MS which can easily reach 300°C so all I needed was an enclosure I could connect to a gas supply and put in there. I considered designing one using CAD and asking my engineer buddies to machine it using their awesome CNC mills. However, that would take time and effort. Instead I found aluminium enclosures for sale from RS that looked perfect and they only cost a few dollars. I drilled a 5mm hole through the side so that I could screw an HPLC compression screw through it. I then passed a piece of 1/16″ stainless HPLC line with a large internal diameter through the screw, added a ferrule and threaded a steel union onto the end on the inside so that the screw head was tight against the outside wall of the box and the union against the inside. I actually had to add a washer to the outside to get a snug seal. Passing gas through the line would now vent into the box through the hole in the union. The seal around the rim of the lid wasn’t particularly tight but it was good enough to allow me to pressurise the chamber with just a few KPa of nitrogen. You can see a video of gas  bubbling out of it underwater here. Here’s the box sitting inside the GC oven:

GC ovens have ready-made ports in the side specifically so that you can add peripherals so I ran the 1/16″ line out of one of these. I had to make a small cut in the outside insulation to pass the line through.

On the outside the 1/16″ line was connected to a 1/4″ piece of HDPE gas line that ran to the regulator on the gas cylinder in my rack. I didn’t use the expensive, instrument-grade nitrogen that I use to run the GCs for this purpose, of course, but cheap industrial-grade oxygen-free stuff. Even better is that I’m not using the ultra high purity helium that the GC-MS runs on either.

Analytical chemists will notice my stupid mistake immediately. I’ve located the box in a GC with a wax column. Wax columns are very useful but they can’t tolerate temperature above 250°C. Derp! Tomorrow I will move it to another GC so I can hit the required temperature. I’m going to try conditioning some cheap, PDMS-packed-tube traps first before I try the expensive stir bars just to make sure it doesn’t destroy stuff.