Data Integrator User's Guide

Contents

Introduction

Video demo

Getting started: performing a simple query

Selecting data sources

Computing item overlap

Configuring output

---

Search the Genome Browser help pages:  

Questions and feedback are welcome.

Introduction

The Data Integrator is a fast and powerful graphical interface that can combine and export data from multiple tracks simultaneously. Like the Genome Browser and Table Browser, it can combine data from the browser database, user custom tracks and track hubs.

Using the Data Integrator, you can retrieve tab-separated text for the data underlying up to five tracks, combining multiple tracks' data for items whose genomic positions overlap the positions of items in the first selected track. Depending on your needs, you may:

• include all columns of all data sources in your query, or only the columns that you select.
• query genome-wide (provided that the results are not so large that the connection times out), or only for your regions of interest.
• view results in your web browser window, or download to a local file, optionally compressed by gzip.

The Data Integrator does not yet have many of the features of the Table Browser tool. For example, if you want to retrieve DNA sequence or GTF format, you will need to use the Table Browser.

Video demonstration of the Data Integrator

   Visit our YouTube channel.

Getting started: performing a simple query

Follow these instructions to perform a basic query that intersects two or more tracks:

Step 1. Pick a genome assembly
Specify the genome assembly from which you'd like to retrieve the data by choosing the appropriate organism using the group and genome menus, then selecting the assembly version from the assembly menu. Note that changing the group menu causes the genome menu to refresh, and changing the genome menu causes the assembly menu to refresh. Assemblies that are uploaded as part of an assembly hub will be available under the hub's name in the group menu.

Step 2. Pick a genomic region
The Data Integrator allows you to choose the genomic regions for which you want output.

• To display all of the data records for the selected track(s), select genome from the region to annotate menu (not available for certain tracks with restrictions on data sharing).
• To restrict the data to a specific position range, select position or search term from the region to annotate menu and type or paste the position into the box. Some examples of specific positions include a chromosome name (chrX), a coordinate range within a chromosome (chrX:100000-400000), or a scaffold name.
• To look up the position range of a genomic element -- such as a gene name, an accession ID, an STS marker, etc. -- or keywords from the GenBank description of an mRNA, select position or search term from the region to annotate menu and then type your search term into the box. For assemblies that have gene annotations, autocompletions of gene symbols are offered. If your search term matches more than one track or item, then a pop-up will appear with matching items and positions; click an item to select its position.
• To query multiple genomic regions, select define regions from the region to annotate menu and then enter up to 1,000 regions in a 3- or 4-field BED file format or as chrom:start-end position ranges.

Step 3. Configure your data sources
The track group menu shows all the annotation track groups available in the selected genome assembly. The names correspond to the groupings displayed at the bottom of the Genome Browser annotation tracks page. When a group is selected from the menu, the track menu automatically updates to show all the annotation tracks available within that group.

• To examine all the tracks available within a certain group (e.g., all gene prediction tracks), select the group name from the track group menu, then browse the entries in the track menu.
• Custom annotation tracks created during the current session are listed under the Custom Tracks group.
• Each connected track hub will be listed as its own group under the track group menu.
• Clicking the "View table schema" link opens a new web browser tab with the Table Browser's schema page which describes the data fields.

After selecting your track from the drop-down menus, click Add. You may add up to five tracks. Some tracks are not available when the region is set to genome due to the data provider's restrictions on sharing. See the Selecting Data Sources section for more details.

Step 4. Rearrange your data sources (Optional)
You can rearrange your data sources by clicking on a track name and dragging it into the desired order. Items are output based on their overlap with the first listed track. The order of the tracks will also determine the order of the fields in the output. Fields from tracks higher in the list will appear first in the output.

Step 5. Choose output fields (Optional)
Click Choose fields... to bring up a checklist of all fields from all selected data sources. By default, every field is selected. You can choose the columns you want in your output by checking the box next to that field. Additionally, you can add and select fields from up to four related tables. See the Configuring Output section for more details.

Step 6. Get output
Click Get output to display the results of the query in the web browser window. The results will be displayed as tab-delimited text. Download results as a file by checking the box next to Send output to file and entering a file name into the box. The output file can also be compressed by gzip.

Selecting data sources

The Data Integrator allows you to select up to five data sources to intersect. These data sources can be tracks provided by UCSC or your own data uploaded using custom tracks or track hubs. Only positional tables, such as BED, wiggle, and genePred, are currently allowed in the output. Other formats such as MAF or BAM are currently unavailable in the Data Integrator.

Items in your selected tracks are output based on their overlap with items in your primary track. The primary track is the one that shows up first in the list of your chosen data sources. In the following example, UCSC Genes would be the primary track and items would be output based on their overlap with items in UCSC Genes:

The order of your selected tracks also affects the order of fields in your output. Fields from tracks that are higher up in the Configure Data Sources box will show up before tracks that are lower in that box. Using the same order of tracks seen in the previous example, this means that fields from the UCSC Genes track will appear first in the output, then fields from DNase Clusters and lastly fields from Conservation - 100 Vert. El. However, your tracks aren't limited to the order in which you selected them, and you can rearrange the order of your tracks at any time. To do so, just click and drag a track to a new position in the list. A track can be removed by clicking the "X" next to it.

Some data providers stipulate that their data can be browsed visually in the Genome Browser, but cannot be provided for download via the Table Browser or Data Integrator. Tracks from those providers do not appear in the track menus. Other providers allow queries within specific regions, but not genome-wide queries. Their tracks appear in the menus, but are grayed out when region is set to genome.

Computing item overlap

The current intersection mechanism the Data Integrator uses is fairly simple. For gene transcripts, mRNAs, and any other tracks where the items have blocks or gaps, the overlap for the entire range of the item is computed. This means that introns and gaps in the item are also considered when computing overlap. This is an improvement upon the Table Browser which only computed overlap with exons. For example, let's consider the following situation using the UCSC Genes and DNase Clusters tracks:

If we were using the Data Integrator to find the DNase peaks that overlapped with the gene in the UCSC Genes track, we would get all of the visible peaks in our output.

Additionally, items are output regardless of the amount of overlap. This means that if there is even a single base of overlap between two items, they are included in the output.

Configuring output

By default, the Data Integrator outputs all fields of all data sources except for the bin column of database tables (a numerical index added by UCSC to speed up database queries). If you are interested in the data from a limited number of fields, you can customize the set of fields to include in the output. After selecting and configuring your data sources as described in the Selecting Data Sources section, click Choose fields... to begin configuring your output. After clicking the button, a pop-up with all the fields from your selected tables will appear:

Uncheck the box next to the field name to exclude it from the output; check the box if you change your mind and want to include it. You can quickly select or deselect all of the fields by clicking Set all or Clear all below the track name.

In the UCSC Genome Browser database, detailed information on the annotations for many tracks is stored in extra tables. This information can include things such as identifiers in other databases, transcript status, or other descriptive information. You can select fields from up to four of these related tables for each of your selected tracks. After clicking Choose fields..., you should see a section labeled "Related tables" below each track. This drop-down menu allows you to select and add related tables to your output.

Note that not all tracks in the Genome Browser database will have related tables. Also, some tables may be unavailable when region is set to genome due to the data provider' s restrictions on sharing. If no "Related tables" section appears for a particular selected track, then there are no related tables for this track.

After you have finished selecting your output fields and related tables, click Done at the bottom of the dialog box to finish configuring your output.

If you want to view your results in the web browser window, simply click Get output. Or, if you want to send your output to a file, check the box next to Send output to file. You can compress your output file using gzip by checking the box next to Compress with gzip (.gz). Then click Get output to begin downloading your file.

Note that for wiggle tracks, the "Value" field is averaged across the region of overlap. In the future, we intend to add an option to output all of the data values in that region of overlap.

If a single item in the primary track has multiple matches in a secondary track, then the Data Integrator outputs multiple rows. Each row repeats the fields of the primary track item, and then adds fields of the nth matching item from each secondary track. If there are multiple secondary tracks, then their items will appear together in a row of output whether or not they overlap each other. For example, note how items overlap (or not) in the three tracks A, B and C in the image below:

Two items from track B (B1, B2) overlap the primary item (A). Three items from track C (C1, C2, C3) overlap the primary item (A). The Data Integrator prints 3 rows of output. B1 and C1, although they do not overlap, appear together in the first row of output. B2 and C2 appear together in the second row of output. The third row of output has empty fields for track B followed by C3's fields. This is the output (when A, B and C are 4-field BED custom tracks; colors added for emphasis):

#A.chrom	A.chromStart	A.chromEnd	A.name	B.chrom	B.chromStart	B.chromEnd	B.name	C.chrom	C.chromStart	C.chromEnd	C.name
chr1	1200	1800	A	chr1	1250	1350	B1	chr1	1425	1525	C1
chr1	1200	1800	A	chr1	1550	1650	B2	chr1	1675	1775	C2
chr1	1200	1800	A					chr1	1700	1800	C3