Managing data freshness
Last updated
Last updated
Maintaining accurate analytics in your map visualizations depends greatly on the freshness of your data. This section will delve into how Builder ensures your data remains current and will detail the options you have for refreshing your data sources.
Builder makes it straightforward to manage data freshness right from the initial map load. Data caching is enabled by default, varying by data type and warehouse provider, but you have the liberty to set specific freshness intervals for your map's data sources.
Default freshness settings
SQL Query sources: By default, data is cached for one year across all connections. If your data remains unchanged, it will be automatically refreshed after a year.
Table sources: The duration of data caching varies depending on your data warehouse provider:
BigQuery and Snowflake: Requests are cached for a minimum of 5 minutes. CARTO continues to serve cached results if the table data hasn't been updated.
Redshift, Databricks, and PostgreSQL: Requests are cached for 30 minutes.
Pre-generated tileset sources: Data is cached for a year. It's automatically refreshed after this period if unchanged.
Customizing data freshness
Choose from predefined freshness periods for your data sources to ensure maps load with the most current data, providing reliable analytics.
Caching plays a crucial role in optimizing the performance and responsiveness of your map. Each component, such as a map layer, leverages its own cache to store data essential for its visualization. Here's how it functions:
Data Storage: When you view a layer within a specific map extent, the system caches the data retrieved by that particular query. This means that if you or another user views the same layer with the same map extent again, the system can quickly display the data from the cache without needing to re-fetch it from the data source.
Handling Changes: Any modifications to the viewport extent, adjustments to widget filters, or changes in SQL parameter inputs trigger a new query to the data warehouse for data that hasn't been cached yet. Once this query is executed, its results are stored in the cache for future use.
This caching mechanism ensures efficient data retrieval and visualization, significantly enhancing the user experience by reducing load times and improving the map's overall performance.
Refreshing your data sources couldn't be simpler. Whether you need to update all sources or just specific ones, Builder's "Refresh" options are designed for efficiency. Initiating a refresh reloads your data sources and their associated layers, clearing any previous cache and sending a new request to your data warehouse. This process guarantees you're always working with the latest data.
Important consideration
Manual refreshes will increase the amount of data processed in your data warehouse, which might have a significant cost associated to it.
The cached objects associated to the data source will be invalidated, and the SQL queries that were executed to generate them will be executed again.
