Importing Data#

Overview#

MsPASS is a data processing system so the first step in using the package for anything is to assemble the data set that you want to process. The focus of our initial development has been tools to assemble data acquired from FDSN (Federated Digital Seismic Network) data centers. Decades ago FDSN adopted the Standard for the Exchange of Earthquake Data (SEED) format. Since then a subset of SEED, commonly called miniseed, has become the universal tool for distributing earthquake data through all FDSN data centers. Miniseed is compressed data format with minimal metadata that has proven useful for data distribution because it creates compact files with lossless compression.

In the past decade FDSN data centers have largely shifted from distributing data by ftp transfers of files to web services. FDSN web services distribute three fundamentally different data types: (1) single channel, sample data delivered as images of miniseed files, (2) station metadata delivered with a standard format called StationXML, and (3) earthquake source data delivered through a standard format called QuakeML. Fortunately for our development efforts a well-developed solution for acquiring all three data types from FDSN sources was already available in obspy. The sections below assume you can consult obspy’s documentation to design the details of how you acquire these data from FDSN data centers. The focus here is how files stored locally from a web service request can be assimilated into MsPASS and validated for processing.

This section also includes a brief discussion of importing data with formats other than SEED/miniseed. Current capability is largely controlled by the extensive set of readers that already exist in obspy. The primary issue you face in importing other formats is handling the metadata name mismatch that is inevitable in dealing with other formats.

FDSN data#

Importing Miniseed Data#

From the MsPASS perspective, at this point in time, the best mechanism to obtain data from the FDSN confederation of data centers is to use obspy. Obspy has two different python functions that can be used to fetch miniseed data from one or more FDSN data centers.

The simplest tool obspy provides is their get_waveforms method of the fdsn web service client. The documentation for that function can be found here. get_waveforms retrieves a relatively small number of waveforms at a time using station codes with wildcards and a time range. It is suitable only for small datasets or as a custom agent to run for weeks acquiring data driven by some large list.
For most MsPASS users the obspy tool of choice is MassDownloader.download. An overview of the concepts of MassDownloader is given here and the detailed docstring for the function can be found here.

The obspy tools are well documented and relatively easy to use. Users are referred to the above pages and examples to build a workflow to retrieve a working data set. We do, however, think it useful to give a few warnings.

Never use get_waveforms to retrieve more than a few thousand waveforms unless you are designing an agent that will run for weeks. It is intrinsically very slow because of the inevitable internet delays that are unavoidable because of the design of that function. Each call to get_waveforms initiates a query request transmitted by the internet to the targeted fdsn web-service server, the server has to do it’s work and respond, a back connection to the caller has to be set up, and then the data transmitted via the internet. The built-in, multiple delays make that process too slow for large data requests.
The authors of obspy developed their MassDownloader because their early experience, like ours, showed get_waveforms was not feasible as a tool to download large data sets. MassDownloader.download makes it feasible to download large data sets. Be warned that feasible, however, does not mean quick. In our experience mass downloads can sustain a transfer rate around a few Mb/s. That is outstanding performance for internet data transfer, but keep in mind 1 Tb of data at that rate will require of the order of one week to download.
A fundamental flaw of the web service model for data downloading is there is no guarantee a request will succeed. We have seen many examples where a large request will have missing data that we know are present at the data center. The reason is that web service is, by design, a one way request with no conversation between the client and server to guarantee success. The authors of obspy’s MassDownloader method seem to have done some tricks to reduce this problem but it is not clear to us if it has been completely solved.

We endorse strongly the following statement in obspy’s documentation on this topic: “Keep in mind that data centers and web services are constantly changing so this recommendation might not be valid anymore at the time you read this. ” In particular, we are aware all the FDSN data centers are in the process of transitioning to a cloud file service model as the next generation of data access. It is a current development effort of MsPASS to provide a simple reader for cloud systems. Examples of using our implementation for S3 on AWS can be found here. As the word “prototype” implies that api is likely to change. Check the docstring api for more recent changes if you are interested in this capability.

Indexing, reading, and optionally converting miniSEED#

Downloading a miniSEED file does not by itself expose its waveforms to MsPASS. The standard local-file import path begins with index_mseed_file. This method scans each file and writes one or more index documents to the wf_miniseed collection. It does not copy or convert the waveform samples; the documents retain the absolute directory, file name, byte offset, byte count, channel codes, and time span needed to read the original file.

For example, this serial loop indexes every .mseed file in one directory:

from pathlib import Path

from mspasspy.db.client import DBClient
from mspasspy.db.database import Database

db = Database(DBClient(), "mydatabase")
data_directory = Path("/absolute/path/to/mseed")

for path in sorted(data_directory.glob("*.mseed")):
    db.index_mseed_file(
        path.name,
        dir=str(data_directory),
        collection="wf_miniseed",
    )

Files should contain miniSEED packets ordered by network, station, location, channel, and time. With the current default segment_time_tears=True, a time discontinuity also starts a new index document. For distributed work, the absolute path recorded by the index must resolve to the same file on every worker that reads it.

An index document can be read as a MsPASS TimeSeries without first rewriting the compressed samples:

query = {}  # Replace with a MongoDB query to select a subset.
for doc in db["wf_miniseed"].find(query):
    datum = db.read_data(doc, collection="wf_miniseed")
    if datum.live:
        # Process datum here, or persist a native MsPASS copy:
        db.save_data(datum, collection="wf_TimeSeries")

read_data uses the stored index to read only the referenced byte range. Calling save_data is optional: use it when the workflow should persist a converted/native copy, and omit it when processing can begin directly from wf_miniseed. Normalization with receiver and source metadata is a separate step described in Normalization.

Warning

In the current implementation, index_mseed_file(normalize_channel=True) computes a channel match after inserting the index records but does not persist the resulting channel_id to those records. Leave normalize_channel=False (the default) and normalize the wf_miniseed collection separately.

Assembling Receiver Metadata#

Receiver metadata from FDSN sources is easily obtained and assimilated into MsPASS using a combination of obspy functions and import functions that are part of the Database class (MongoDB handle) of MsPASS.

FDSN receiver metadata is obtainable through one of two fundamentally different approaches: (1) the older “dataless SEED” file format, and (2) the XML-based format transmitted through web services called StationXML. Both are now FDSN standards. It is our opinion that dataless SEED is archaic and we have not devoted development time to supporting the format. A key reason is that there are multiple, existing solutions to reading dataless SEED files. Most notably dataless SEED files can be theoretically be read and translated with obspy as a format option to the read_inventory function we discuss in more detail below. We have not tested that approach, however, and would recommend the much simpler and cleaner StationXML format.

We recommend users utilize obspy to assemble receiver metadata from FDSN data centers. There are two different tools obspy provides. We have found both are usually necessary to assemble a complete suite of receiver metadata.

The fdsn client has a method called get_stations that is directly comparable to get_waveforms. It uses web services to download metadata for one or more channels of data. It has a set of search parameters used to define what is to be retrieved that is usually comprehensive enough to fetch what you need in no more than a few calls. Be warned it retrieves the results into memory into a custom obspy data object they call an Inventory. The docstring for an Inventory object can be found here. An Inventory object can be viewed as more or less a StationXML format file translated into a python data structure with a few added decorations (e.g. plotting). We return to this point below when we discuss how these data are imported to MsPASS.
When you use MassDownloader.download you have the option of having that function download the station metadata and save the actual StationXML data files retrieved from web services. The resulting files can then be read with their read_inventory method described here.

Both of the approaches above can be used to create an obspy Inventory object in python: for (1) that is the return of the function while for (2) it is the output of a call to read_inventory with format="STATIONXML". We use the obspy Inventory object as an intermediary for storing receiver metadata in a MongoDB database. The Database class has a method we call save_inventory described here. That method translates an Inventory object into documents stored in what we call the channel and site collections. As noted many other places in our documentation channel contains receiver metadata from TimeSeries data while site contains a more subset of the same information more appropriate for Seismogram data. A typical application of save_inventory can be seen in the following code framgment extracted from our tutorials:

from mspasspy.db.database import Database
from mspasspy.db.client import DBClient
dbclient=DBClient()
db=Database(dbclient,'getting_started')
inv=client.get_stations(network='TA',starttime=starttime,endtime=endtime,
                    format='xml',channel='BH?',level='response')
ret=db.save_inventory(inv,verbose=False)
print('save_inventory returned values=',ret)

As noted above an Inventory object is more or less an image of a StationXML file. StationXML is complete, but often contains a lot of baggage that is not necessary for most workflows and would unnecessarily bloat a MongoDB database. For that reason, in MsPASS we do not extract the entire contents of the StationXML file image. As noted in the documentation for save_inventory we save receiver locations, component orientations, and a serialized version of the response data. If your application requires additional data from the StationXML image you will need to extract that information from the Inventory object and use the update functions of MongoDB to add what you need. As noted many times in this manual MongoDB is completely cavalier about what is stored in any given document so adding additional key-value pairs will not break any MsPASS algorithms.

Source Metadata#

Source metadata is a vastly more complicated problem that receiver metadata. The following is a litany of the complexity we needed to deal with in a generic framework like MsPASS that could support all forms of data seismologists deal with.

What defines source metadata is as wildly variable as anything we can think of. Some methods like noise correlations or studies of noise do not require any source information. Even when source information is required the attributes required are not fixed. Some data require only coordinates, but the coordinates may be geographic or some local coordinate system. Some, but not all data need moment tensor estimates. The list continues. The complete flexibility of MongoDB in defining what attributes are loaded as the source “document” effectively solves this problem.
With some data there is one and only one source estimate for each datum. The type example is seismic reflection data where the shot coordinates are defined with standard “geometry” attributes. Natural source data often have multiple, competing estimates of source metadata for the same “event”. The CSS3.0 schema, for example, handles this issue by defining two relational database tables called event and origin with the concept that an event is a unique source while an origin is one of multiple possible source estimates for a given event. Although the flexibility of MongoDB could provide a workable solution for the multiple origin problem (the likely solution would involve subdocuments) we chose to not add that complexity to MsPASS. At present we assume that when using the source collection to define source metadata a given waveform will be associated with one and only one source document.
There are large variations in the complexity of the problem of associating a seismic datum to a set of (document) source metadata. That problem is trivial with seismic reflection data compared to most natural source data. Until recently all seismic reflection data was naturally collected as “common shot (source) gathers”. Most seismic reflection geometry definitions simply require an ordered list defines the order of gathers in a linear data file. The same issue is much more complex with passive recording. A partial list includes: (1) irregular sample rate, (2) irregular start times, (3) there may or may not be a need to compute or use a set of phase arrival times, and (4) overlapping, duplicate copies of the same data in multiple input data files. Because of the complexity of this problem we provide only a partial set of tools for associating waveforms with source data: MongoDB normalization described in the section of this user’s manual titled Normalization.

MsPASS currently supports directly only one mechanism for loading source metadata. That method is a similar in approach to the way we handle FDSN station data. That is, we use obspy for the machinery to download the data from FDSN web services and translate the obspy python data structure, which in this case is called a Catalog, into MongoDB source documents.

Like the receiver problem, obspy has two comparable functions for retrieving source metadata.

get_events is an obspy function that is very similar to the receiver equivalent get_stations noted above. Their documentation on this function can be found here. Like the receiver equivalent it has search criteria to yield a set of source data based on some spatial, time, magnitude, and/or other criteria. In addition, like get_stations, get_events returns the result in a python data structure that in this case they call a Catalog. The Catalog class is more or less an image of the FDSN standard for web service source data in XML format they call QuakeML. The biggest issue with this approach for many workflows is that it is too easy to create a collection of source data that is much larger than the number of events actually in the data set.
The read_events function reads a saved event file, including QuakeML, into a Catalog. MassDownloader.download does not create QuakeML files; query the event service separately with get_events and optionally save that catalog as QuakeML.

The procedure to load source data for a MsPASS workflow derived from one of the obspy methods is comparable to that described above for FDSN StationXML data. That is, we use an obspy python data structure as the intermediary for the import. get_events returns the obspy Catalog class directly while local QuakeML files are read with the obspy function read_events described here. A Catalog instance can then be saved to a MongoDB source collection using the Database method called save_catalog. The following is a complete example using a local QuakeML file.

from obspy import read_events

catalog = read_events("events.xml",format="QUAKEML")
number_saved = db.save_catalog(catalog,verbose=False)
print("source documents saved=",number_saved)

Be aware that save_catalog inserts every event without checking for duplicates and requires each event to have a preferred origin and preferred magnitude.

An alternative for which we provide limited support is importing catalog data from an Antelope database. We have a prototype implementation in the module mspasspy.preprocessing.css30.dbarrival but emphasize that code is a prototype that is subject to large changes. The actual dbarrival.py prototype will almost certainly eventually be depricated. We include it in our initial release as a starting point for users who may need this functionality. The approach used in the prototype is independent of the relational database system used for managing the source data. That is, the approach is to drive the processing with a table defined as a text file. The same conceptual approach could be used as the export of a query of any relational database that is loaded internally as a pandas dataframe.

Importing Other Data Formats#

Currently MsPASS depends completely on obspy for importing waveforms in a format other than miniseed. The following is pseudocode with a pythonic flavor that illustrates how this would be done for a list of data file to be processed:

from obspy import read
from mspasspy.db.database import Database
from mspasspy.db.client import DBClient
from mspasspy.util.converter import Stream2TimeSeriesEnsemble
dbclient=DBClient()
db=Database(dbclient,'mydatabasename')
   ...
for fname in filelist:
  st = read(fname,format="SOMEFORMAT")
  d = Stream2TimeSeriesEnsemble(st)
  db.save_data(d)

where SOMEFORMAT is a keyword from the list of obspy supported formats found here and Stream2TimeSeriesEnsemble performs the generic one-Trace-to-one- TimeSeries conversion. For formats whose headers require special treatment, wrap it in a format-specific converterfunction. That function needs to handle the idiosyncrasies of how obspy handles that format and convert the stream st to a TimeSeriesEnsemble using the MsPASS converter function Stream2TimeSeriesEnsemble documented here. That is a necessary evil because as the authors of the obspy write in their documentation some formats have concepts incompatible with obspy’s design. Although we cannot provide unambiguous proof we have confidence the same is not true of MsPASS because the TimeSeries container is more generic than those used in obspy as we discuss in the section Data Object Design Concepts.

There are two different issues one faces in converting an external format to the instance of an implementation of a seismic data object like TimeSeries or TimeSeriesEnsemble:

The sample data vector(s) may require a conversion from various binary structures to the internal vector format (in our case IEEE doubles). The approach we advocate here solves that problem by not reinventing a wheel already invented by obspy. If you face the need to convert a large quantity of data in an external format it may prove necessary to optimize that step more than what obspy supplies as we have no experience on the efficiency of their converters. Don’t enter that gate to hell, however, unless it is essential as you may face a real-life example of the Dante quote: “abandon hope all ye who enter here”.
Every format has a different header structure with few, if any, overlaps in the namespace (i.e. the key-value pair defining a concept). That means both the string used in the api to refer to an attribute and the type of the value. The obspy readers handle this issue differently for different formats. That variance is why we suggest any conversion will require developing a function like that we call converterfunction above.

The MsPASS schema class (see this page for details) has tools we designed to aid conversion of Metadata (i.e. item 2 above) from external representations (format) of data to MsPASS. In particular, the apply_aliases and the inverse clear_aliases were designed to simplify the mapping for key-value pairs in one namespace to another. To utilize this feature for a given format you can either create a yaml file defining the aliases or hard code the aliases into a python dict set as key:alias.

We close this section by emphasizing that at this time we have intentionally not placed a high priority on development of complete tools for importing formats other than SEED/miniseed. We consider this one of the first things our user community can do to help expand MsPASS. If you develop an implementation of one of the functions we gave the generic name converterfunction above we encourage you strongly to contribute your implemetation to the MsPASS repository.

Validating an Imported Data Set#

After importing any data to MsPASS (miniseed included but especially any specialized import function output) you are advised strongly to run the MsPASS command line tool mspass-dbverify on the imported collection. We advise you run both the required test (-t required) and the schema_check test (-t schema_check) before running a significant workflow on an imported data set.

For example, run the two tests separately on a native waveform collection:

mspass-dbverify mydatabase -c wf_TimeSeries -t required
mspass-dbverify mydatabase -c wf_TimeSeries -t schema_check