Data Observatory

Data Observatory

Visualizing Data Observatory datasets

Creating maps

Data Observatory datasets can be visualized from the CARTO Workspace using Builder. You can easily do so by clicking on the Create map action in the subscription’s detail page, available from the Data Observatory section of the Data Explorer:

Using Builder to create a map from Data Observatory data.
Data Observatory example map in Builder.

Or by adding a new Data Observatory source to an existing map:

Adding a Data Observatory subscription to an existing map

Those datasets whose size is within platform limits will be visualized in full. Bigger datasets will be applied a spatial filter (a buffer around the centroid of the most populated city of the dataset’s country), but this filter can be modified at your own will through the provided SQL query (check out this tutorial for more details). These datasets will require a tileset to be visualized in full. Please refer to the Creating Data Observatory tilesets section to learn more.

Data Observatory example map in Builder.

Choosing a connection

You will be asked to select the connection that will be used to create a map with your Data Observatory subscription. The chosen connection will be used to retrieve the necessary data to add the layer to the map. Currently, CARTO Data Warehouse, BigQuery and Snowflake connections are supported; Redshift and Databricks support is coming soon.

Choose a connection to create a map with your Data Observatory subscription.

In order to be able to use a Snowflake connection to create a map, the data first needs to be imported into your database. This import process is performed by our engineering team on a request basis.

To request it, go to the subscription’s page, click on the Create map button and choose the desired Snowflake connection. You will be asked to request access to the dataset.

Request to access your subscriptions in Snowflake.

Once we receive your request, we will get in touch with you to coordinate the import process. The data will be imported into a schema called CARTO that will be created in the Snowflake database you have set up in your Snowflake connection. Finally, you will be able to create a map using such connection.

Create map of your Data Observatory subscription using a Snowflake connection.

Creating tilesets

Some of the spatial datasets offered in the Data Observatory are massive (a few TB), either due to their global coverage, such as WorldPop or NASADEM, or their fine granularity, such as ACS Sociodemographics at census block group level, and their visualization requires the creation of a tileset.

Multiresolution quadkeys
NASADEM worldwide elevation tileset.

From the Workspace

When a Data Observatory subscription requires a tileset to be visualized in full, a Create tileset option will be available from the subscription’s page.

Create tileset menu option.

Clicking on this option will display a wizard that will guide you through the tileset creation process:

  1. Choose the name and location of the output tileset. Please note that you should have writing permissions in the destination dataset.
Create tileset wizard: step 1.
  1. Select the zoom range for which the tileset will be created.
Create tileset wizard: step 2.
  1. Select the attributes (data columns) that will be included in the tileset.
Create tileset wizard: step 3.
  1. Launch tileset creation.
Create tileset wizard: step 4.

Once the tileset creation has been launched, you can check its status using the processing jobs menu available at the top right of the page.

Create tileset job progress.

Finally, once the tileset is created, you can add it as a layer to any of your maps as with any other tileset. Please refer to this guide for more information.

From your data warehouse

You can also create tilesets directly from your data warehouse using SQL commands. This option is currently only available for BigQuery using any of the available procedures in the Analytics Toolbox. You can create two types of tilesets:

  • Simple: choose this type if you are wanting to visualize the original features of the dataset. To get started, find here a set of examples on how to create simple tilesets using the CREATE_TILESET procedure.
  • Aggregation: choose this type when your dataset is composed of points and you want to see them aggregated. To get started, find here a set of examples on how to create aggregation tilesets using the CREATE_POINT_AGGREGATION_TILESET procedure.

Both of the aforementioned procedures take as input a SQL query specifying the data that you want to transform into a tileset. If the case of Data Observatory datasets, you can use the example query provided as part of the Access in functionality (learn more here). We recommend that in order to minimize the cost and avoid reaching BigQuery internal limits, you only include in your input query the data columns that you strictly need for your visualization.