Release notes
October 28th, 2024 (v1.17.0)
Module data
Fix
Fixed a problem with DATAOBS_ENRICH_POLYGONS_WEIGHTED and ENRICH_POLYGONS_WEIGHTED that caused an issue when min, max, or count were used as aggregation methods for weight variables.
Module lds
Improvement
Added async batch routing services support for TomTom, providing a 2x-3x preformance improvement in processing speed.
Module cpg
Fix
Modify RUN_CUSTOMER_SEGMENTATION procedures to overwrite internal temp tables and avoid crashing when called twice in the same session.
Module retail
Fix
Modify COMMERCIAL_HOTSPOTS, FIND_TWIN_AREAS, FIND_TWIN_AREAS_V1 (discontinued version), FIND_TWIN_AREAS_WEIGHTED and FIND_TWIN_AREAS_WEIGHTED_V1 (discontinued version) procedures to overwrite internal temp tables and and avoid crashing when called twice in the same session.
Module statistics
Fix
Modify AREA_OF_APPLICABILITY, ORDINARY_KRIGING_TABLE, SMOOTHING_MRF_H3, SMOOTHING_MRF_QUADBIN and SMOOTHING_MRF_QUADINT (discontinued version) procedures to overwrite internal temp tables and avoid crashing when called twice in the same session.
September 24th, 2024 (v1.16.0)
Module tiler
New
Add
tile_resolution
to all tileset creation options. This new setting allows controlling the size of the tiles when rendered in a map. Higher values produce larger tiles.
Module statistics
New
Add DETECT_SPATIAL_ANOMALIES and DETECT_SPACETIME_ANOMALIES procedures for detecting anomalous space-time regions in the data. This new capability allows for uncovering unusual patterns in historical spatial or space-time data and for accurately identifying emerging anomalous patterns in space and time.
Fix
Modify GETIS_ORD_SPACETIME_QUADBIN_TABLE and GETIS_ORD_SPACETIME_H3_TABLE procedures to overwrite internal temp tables and not crash when called twice in the same session.
Module telco
Fix
Fix input format of
terrain_raster_band
in TELCO_PATH_PROFILE_RASTER procedure.
August 23rd, 2024 (v1.15.1)
Module statistics
Fix
Fixed an issue with GETIS_ORD_SPACETIME_H3 and GETIS_ORD_SPACETIME_QUADBIN procedures that were returning wrong results when “time_freq” was passed as lower case.
July 2nd, 2024 (v1.15.0)
Module lds
Fix
Fixed a bug where there was more than one matching row in the target for each row in the source.
GEOCODE_REVERSE_TABLE
now useST_DISTINCT
to solve this issue
Module raster
New
Adjustments to use only full-resolution data when the raster table has overviews (blocks of resolution smaller than the block_resolution stated in the metadata)
We can now specify per-band no-data values for raster datasets. Rasters bands also now have a ‘
colorInterp
’ metadata attribute with possible values of red, green, blue, alpha or unknown.
Improvement
Optimized raster procedures (
RASTER_VALUES
,RASTER_AGG_VALUES
) to manage bigger data sources. Additionally, the optionintersect_center
is supported in both procedures to use the centroid of the pixel instead of the contour, in order to speed up the extraction/aggregation for relatively big polygons.New options in
RASTER_AGG_VALUES
:groupby_vector_columns
andgroupby_raster_columns
to have finer control over the aggregation.
Module import
Fix
Import functions in AT Gateway now raise an exception when there is an error. This replaces the previous status message.
Module statistics
New
New TIME_SERIES_CLUSTERING procedure, which expects a table containing time series (indexed by a time step column and a unique series identifier) and can group the time series in as many clusters as the user requests.
New SPACETIME_HOTSPOTS_CLASSIFICATION procedure. This procedure is designed to analyze spatio-temporal data in order to identify and categorize locations based on their hotspot or coldspot status over time. Utilizing z-score values generated by the Space-Time Getis-Ord function.
Module telco
New
New
TELCO_PATH_PROFILE
procedure, that finds the path between the transmitter (Tx) and receiver (Rx), the obstacles that exist along the propagation path, i.e. those that intersect with the propagation line, and then it evaluates the impact of each obstacle's heightNew
TELCO_PATH_PROFILE_RASTER
procedure, an optimised version of the nominal TELCO_PATH_PROFILE procedure which only accepts as inputs a clutter and a terrain elevation rasters loaded using CARTO raster loader or Import API.New
CLOSE_IN
: This procedure implements a statistical propagation model that describes large-scale propagation path loss over distance at all relevant frequencies in a certain outdor scenario.New
EXTENDED_HATA
: This new procedure implements the empirical propagation model Extended HATA for radio propagation path loss for the range of frequencies 30 MHz - 3000 MHz in an outdoor scenario.
Module quadbin
Improvement
Improved QUADBIN_POLYFILL_MODE and QUADBIN_POLYFILL_TABLE performance with big polygons on high-resolution cells.
May 22nd, 2024 (v1.14.0)
Module http_request
New
We have a new module named “http_request”. It allows making requests to external endpoints through the function
HTTP_REQUEST
Module import
New
We have a new module named “import”. It allows importing tables for a given URL containing a file in any of the formats supported by the CARTO Import API. The procedure in charge of this import is
IMPORT_URL
.
April 18th, 2024 (v1.13.0)
Module raster
New
Added new
RASTER_VALUES
andRASTER_AGG_VALUES
procedures:RASTER_VALUES
obtains pixel values from a raster where it overlaps with an optional vector data input. It accepts SQL expressions, including operations between bands.RASTER_AGG_VALUES
obtains aggregated values from a raster's pixel values where it intersects with a vector data input. It accepts any SQL aggregation expression.
Module clustering
Fix
Update
ST_CLUSTERKMEANS
in order to avoid generating fewer clusters than requested. The maximum number of clusters now will be equal to the number of different points passed.
Module lds
Fix
GEOCODE_TABLE
not working properly for “geom_column
” parameter's value different from “geom”.
March 21st, 2024 (v1.12.0)
Module lds
New
All external functions (including those used by the procedures in the lds module) now use a single, unified endpoint. This also simplifies the manual installation of the Analytics Toolbox.
GEOCODE_TABLE now uses async batch geocoding with TomTom, boosting the geocoding speed considerably. Also matching by rows is applied ensuring that the address that reaches the TomTom endpoints is as similar as possible as the table value.
Module data
New
Add ENRICH_POLYGONS_WEIGHTED procedure. This new procedure enables users to enrich polygons with a provided data query by taking into account the ratio of a specified feature between the intersected and original areas in a way to weigh the contribution of each feature in the enrichment. Note that the original methods only use the geographic area of intersection for this weight.
Module statistics
Fix
Fix output table schema on the SQL Reference for the procedures CREATE_SPATIAL_COMPOSITE_SUPERVISED and CREATE_SPATIAL_COMPOSITE_UNSUPERVISED.
Add support in the CRONBACH_ALPHA_COEFFICIENT procedure for supporting ‘
NUMERIC
’, ‘BIGNUMERIC
’.
Module random
Fix
ST_GENERATEPOINTS now guarantees to return the exact number of points requested.
February 16th, 2024 (v1.11.1)
Module retail
Fix
Fixed aggregation column name suffix in BUILD_CANNIBALIZATION_DATA.
Module statistics
Fix
Set null weights to 0 to standarize results in all Moran's I and Local Moran's I.
Add warning to Moran's I procedures when no cells have neighbors.
January 18th, 2024 (v1.11.0)
Module quadbin
New
New QUADBIN_DISTANCE function that returns the distance (number of cells) between two quadbin cells.
Fix
Improved precision when calculating cell indexes from latitudes close to the poles (±90º).
Module h3
Fix
Fixed an issue with the H3_POLYFILL_TABLE procedure that was not working due to a broken function reference.
Module transformations
New
New ST_POINTONSURFACE function that returns a point contained within a given polygon.
Module tiler
Fix
Support adding properties from BIGNUMERIC columns in the CREATE_TILESET procedure.
Module data
Fix
Fixed an issue when using AVG aggregation, that was not correctly computed when “data_query” contained NULL values.
Fixed a problem with DATAOBS_ENRICH_POINTS, that was not parsing vars metadata correctly and producing a crash.
Fixed a problem when using “Default” aggregation in DATAOBS_ENRICH_* procedures.
Module lds
Fix
Fixed a problem that caused GEOCODE_REVERSE_TABLE crash when a table contains multiple points very close to each other.
Module raster
Improvement
Support COUNTIF aggregation method in RASTER_VALUE and RASTER_VALUE_TABLE procedures.
Module statistics
Fix
Added a normalization constant for MORANS_I_* procedures
December 21, 2023 (v1.10.0)
Module raster
New
New
RASTER_VALUE
, andRASTER_VALUE_TABLE
procedures: updated raster format, introduced performance optimizations and support for custom operations, including multi-band and aggregation expressions.Raster loader v0.4.0: supports new raster format, upload multiple bands and performance optimizations.
Deprecated
Previous versions of
RASTER
procedures in beta
Module retail
Improvement
Updated
FIND_TWIN_AREAS
procedures to also work with categorical data.The previous version of the Twin Areas procedures are still available as:
Module cpg
Improvement
Updated
FIND_SIMILAR_LOCATIONS
procedures to also work with categorical data.The previous version of the Twin Areas procedures are still available as:
Module tiler
Fix
Fixed a bug in CREATE_POINT_AGGREGATION_TILESET procedure that was causing issues for generating higher zoom level tiles.
Module data
Improvement
Updated
GRIDIFY_ENRICH
procedures to also work with categorical data.The previous version of the procedure is still available as
GRIDIFY_ENRICH_V1
Deprecated
Old quadint/quadkey support in all procedures
November 13, 2023 (v1.9.0)
Module lds
New
Add
SETUP
procedure. This procedure helps configuring the Analytics Toolbox in custom installations. Check this documentation about installing the Analytics Toolbox in your own BigQuery project for more information.Add LDS functions:
GEOCODE
,GEOCODE_REVERSE
,ISOLINE
. This functions complement the already existing procedures. They allow geocoding a single address or getting a single isoline without creating a result table.Replace
LDS_QUOTA_INFO
procedure withGET_LDS_QUOTA_INFO
function. This new function allows checking your currently consumed quota with a single query, without the need to store the result as a table.
Fix
Correctly handle addresses with quotes in
GEOCODE_TABLE
.
Module data
Change
Change location of spatial catalog tables: This is only an internal change. Now the catalog is hosted along with the Data Observatory subscriptions. This change decouples the Analytics Toolbox location from the Data Observatory location, providing more flexibility for future developments.
Module statistics
Fix
Use temporal tables to avoid existing issues in
GETIS_ORD_SPACETIME
.
Module retail
Fix
Fix bugs in kring decay procedures:
KRING_DECAY
,KRING_DECAY_GRID
. These procedure are used internally by other procedures. They work finding cells that are within the K-Ring of each cell in a grid, and assigns a weight based on the distance. This helps model changes in a variable across neighboring cells.\
Module cpg
Change
Adapt composite score supervised procedures: The algorithm used for computing supervised composite scores has been changed from
BOOSTED_TREE_REGRESSOR
toRANDOM_FOREST_REGRESSOR
to avoid overfitting.
October 13, 2023 (v1.8.0)
Module statistics
New
We have added new PCAMIX procedures:
BUILD_PCAMIX_DATA: to create the data used by the PCAMIX method
BUILD_PCAMIX_MODEL: to train a PCA model using the PCAMIX method
PREDICT_PCAMIX_SCORES: to predict the PC scores on the trained/new data
Module retail
Improvement
Improved performance in the COMMERCIAL_HOTSPOTS procedure.
Module lds
Improvement
New batch service for CREATE_ISOLINES. Used automatically by the procedure. No changes required.
September 8, 2023 (v1.7.0)
Module statistics
New
Added
_TABLE
procedures for some statistic functions. The_TABLE
version of the procedures expect a table as input, overcoming performance limitations of the functions that receive an array as input:
Module retail
Fix
Fix default options for BUILD_REVENUE_MODEL that were causing errors when using linear regression models
New
BUILD_REVENUE_MODEL now supports random forest models
Module lds
Fix
Fix
GEOCODE_TABLE
not iterating through input table rows.
August 7, 2023 (v1.6.0)
Module cpg
New
We have added the new procedures CREATE_SPATIAL_SCORE and CREATE_SPATIAL_PERFORMANCE_SCORE to allow users create scores to analyze their network of merchants/point-of-sale based on a business performance metrics or geospatial attributes to analyze their suitability for the CPG brands.
Module statistics
Fix
We have fixed order of return values in LOCAL_MORANS_I_H3 and LOCAL_MORANS_I_QUADBIN functions.
Module h3
New
We have added a new function H3_POLYFILL_MODE that enables different polyfill modes: center, intersects, contains.
Improvement
We have improved the performance of H3_POLYFILL by an average factor of 60x.
Module quadbin
New
We have added a new function QUADBIN_POLYFILL_MODE that enables different polyfill modes: center, intersects, contains.
Improvement
We have improved the performance of QUADBIN_POLYFILL by an average factor of 10x.
July 14, 2023 (v1.5.0)
Module lds
New
We are now allowing users to send any of the parameter options supported by the isolines service provider in CREATE_ISOLINES.
We have improved our logging of results in GEOCODE_TABLE so more information about processed and geocoded rows is provided to the user. Also the new option “carto_force_geocode” allows the re-geocoding of already geocoded rows.
Fix
We have applied some fixes to the CREATE_ROUTES function so the metadata column can be successfully created and informed.
Module data
New
We have added the new procedures DATAOBS_ENRICH_POLYGONS_WEIGHTED and DATAOBS_ENRICH_GRID_WEIGHTED. They allow users to enrich polygons or grids based on spatial indexes (e.g. H3, Quadbin) with features from Data Observatory subscriptions by taking into account the ratio of a specified feature between the intersected and original areas in a way to weight the contribution of each feature in the enrichment. Note that the original methods only use the geographic area of intersection for this weight.
Fix
We have updated the GRIDIFY_ENRICH procedure so the default value for the “decay” function is taken into consideration.
Module retail
New
We have added the possibility of passing custom polygons as trade areas in BUILD_CANNIBALIZATION_DATA and CANNIBALIZATION_OVERLAP.
Fix
We added exceptions to both FIND_TWIN_AREAS and FIND_TWIN_AREAS_WEIGHTED procedures to ensure that the parameter “origin_query” includes only one row (the origin cell).
Module statistics
New
We are now allowing to use Numeric and Bignumeric as predictors data types types in the procedures: AREA_OF_APPLICABILITY, ENV_BLOCKING and CREATE_SPATIAL_COMPOSITE_SUPERVISED.
Fix
We modified the bucketization options in CREATE_SPATIAL_COMPOSITE_SUPERVISED and in CREATE_SPATIAL_COMPOSITE_UNSUPERVISED to better fit potential use cases covered by each procedure.
Module quadbin
Change
We have updated the formula used in QUADBIN_FROMLONGLAT so it gets standardized. In addition a new implementation of QUADBIN_TOCHILDREN has been added in order to increase performance.
June 6, 2023 (v1.4.0)
Module lds
New
We have created a new procedure named CREATE_ROUTES, which calculates the routes (returned as linestrings) between a set of given origins and destinations (points) in a query. It supports different transport modes and advanced options as the other functions of the lds module.
Module statistics
New
We have added new functions in the statistics module to perform space-time cluster analysis, namely GETIS_ORD_SPACETIME_QUADBIN for quadbin indexes and GETIS_ORD_SPACETIME_H3 for H3 indexes.
Fix
We have changed the input type and defined default values for some internal parameters that affect CREATE_SPATIAL_COMPOSITE_SUPERVISED and CREATE_SPATIAL_COMPOSITE_UNSUPERVISED. Also added some missing brackets to them.
Changed code to improve scalability in CREATE_SPATIAL_COMPOSITE_UNSUPERVISED.
Renamed internal function names to avoid possible conflicts in the future.
Module cpg
Fix
We have introduced a better way to handle large tables and drop intermediate tables in the UNIVERSE_MATCHING function.
Module tiler
New
We have added metadata_partition option to CREATE_TILESET, CREATE_SIMPLE_TILESET, CREATE_POINT_AGGREGATION_TILESET, CREATE_SPATIAL_INDEX_TILESET. A metadata_partition label is added to the table when the partition is not NULL. This optimizes tilesets for its use within CARTO.
May 5, 2023 (v1.3.0)
Module statistics
New
Set of procedures to compute Area of Applicability of a model built with BigQuery ML:
AREA_OF_APPLICABILITY: This procedure computes the Area of Applicability (AOA) of a Bigquery ML model. It generates a metric which tells the user where the results from a Machine Learning (ML) model can be trusted when the predictions are extrapolated outside the training space (i.e. where the estimated cross-validation performance holds).
ENV_BLOCKING: This procedure derives cross validation (CV) folds based on environmental blocking.
Fix
We have added a new mandatory parameter called correlation_var when scoring_method = FIRST_PC in the CREATE_SPATIAL_COMPOSITE_UNSUPERVISED
We have improved the management of temporary tables in all procedures to compute spatial composites (scores)
Module raster (beta)
Improvement
We have released a new version of the raster-loader (v0.3.1), with the following improvements:
Store ‘nodata’ value in metadata
Add resolution level to metadata
Fix bands metadata aggregation
We allow for the use of ‘nodata’ metadata in RASTER_ST_GETVALUE
We have released experimental rendering capabilities for the raster data in deck.gl: https://deck.gl/docs/api-reference/carto/carto-layer#raster-data-experimental
April 4, 2023 (v1.2.0)
Module statistics
New
We have released a new set of procedures within the statistics module to enable users to create spatial scores (also known as composite indicators or indexes) derived from a combination of different features. We have included 3 different procedures:
CREATE_SPATIAL_COMPOSITE_SUPERVISED: to compute a spatial composite score as the residuals of a regression model which is used to detect areas of under- and over-prediction.
CREATE_SPATIAL_COMPOSITE_UNSUPERVISED: to perform an aggregation of individual variables, scaled and weighted accordingly, into a spatial composite score.
CRONBACH_ALPHA_COEFFICIENT: to measure the internal consistency of the variables used to derive the spatial composite score.
Module lds
Change
We have updated the LDS module to be aligned with the new usage quota model, working as an annual budget and combining consumption across all supported LDS services.
Fix
GEOCODE_TABLE now supports null values in the input address column, ignoring them when running the geocoding process to provide results only for the informed addresses.
March 7, 2023 (v1.1.0)
Module raster (beta)
Improvement
Note
The raster module of the Analytics Toolbox for BigQuery v1.1.0 will work only with raster-loader v0.3.0, which pre-computes and stores metadata in the raster table and homogenizes the generic/quadbin raster table format. So, if you update your Analytics Toolbox version to the most recent v1.1.0, please ensure you are using the raster-loader v0.3.0.
We have made the RASTER_GETMETADATA faster by retrieving metadata from the table and avoiding creating again the original table.
We have improved the RASTER_ST_GETVALUE by removing pixel_geog column and cluster results by quadbin in the quadbin raster format.
We have enhanced the RASTER_ST_GETVALUE_FROM_TABLE outcome by preserving the input columns, renaming centroid_quadbin_block and centroid_quadbin_pixel to block_quadbin and quadbin, clustering the results by quadbin in quadbin raster and removing few non-informative columns (e.g. geog and myoffset).
Module cpg
New
We have released a new set of functions to enable users to perform the merchant universe matching analysis. It includes 2 different functions, UNIVERSE_MATCHING which performs a fuzzy match between two different POI datasets based on location and name similarity (i.e. one dataset should include current merchants and the other one the complete universe of all potential merchants), and UNIVERSE_MATCHING_REPORT that generates report-like tables summarizing market penetration based on the results of the UNIVERSE_MATCHING analysis.
Fix
We have implemented a fix in the CUSTOMER_SEGMENTATION_ANALYSIS_DATA.
Module lds
Improvement
From this version, we support line break (\n) and other escaped sequences in addresses in GEOCODE_TABLE.
January 31, 2023 (v1.0.0)
Raster module (beta)
New
Note that Analytics Toolbox for BigQuery v1.0.0 will work only with raster-loader v0.2.0, to use the latest version of rater-loader please ensure you also have access to the latest version of the Analytics Toolbox
New module to operate with raster data in BigQuery. This module includes the procedures RASTER_ST_GETVALUE and RASTER_ST_GETVALUE_FROM_TABLE to obtain the values of the raster in one or multiple geographies, respectively. It also includes the RASTER_GETMETADATA procedure to obtain metadata associated with the raster file.
Routing module (beta)
Improvement
New enhanced version of the routing module with procedures ROUTING_MATRIX to calculate origin-destination matrices and ROUTING_ISOLINES to compute isolines around a set of locations, both supporting multiple transportation modes (car, bike, and walk). These new functions run on top of CARTO’s road network (derived from OSM segments) that is available as a public subscription in the Data Observatory. Please note that these improvements imply breaking changes with the previous version of the routing module.
Module retail
Fix
Fix in CANNIBALIZATION_OVERLAP procedure / example.
Fix in TWIN_AREAS_WEIGHTED when sum is 1.
Module statistics
New
Add IDW function (Inverse Distance Weighted).
Module tiler
Improvement
Update default zoom values: geo, quadbin, h3 (12, 12, 6).
Fix spatial index tiler with empty properties.
December 22, 2022
Module lds
Improvement
We have added the possibility to configure more options as parameters when executing the functions to CREATE_ISOLINES. These new options, which depend on the LDS service provider, allow the user to configure more transportation modes such as truck or bike, the possibility of specifying departure or arrival times allowing the creation of reverse isolines, and other options like different routing modes. Please note that these improvements imply breaking changes with the previous version of the CREATE_ISOLINES functions.
Added new confidence/relevance metadata to the results of the geocoding function GEOCODE_TABLE. Please note that this improvement has implied breaking changes with the previous versions of the GEOCODE_TABLE and GEOCODE_REVERSE_TABLE functions.
Module cpg
New
New FIND_SIMILAR_LOCATIONS function to the CPG module, that allows users to identify which locations (e.g. merchants, stores) are more similar to a chosen location (e.g. top performant) based on the characteristics of their surrounding areas (or trade areas).
Module retail
Improvement
Update in BUILD_CANNIBALIZATION_DATA and CANNIBALIZATION_OVERLAP to support buffer, kring and isoline methods. Please note that these improvements have implied some breaking changes in the aforementioned functions invalidating their previous version.
Module data
Fix
We have implemented a fix in the weighted averages computation in the GRIDIFY_ENRICH procedure.
Module quadbin
New
Quadbin/quadkey conversion functions have been added to ease the conversion from quadbin to quadkey and from quadkey to quadbin indexes.
November 15, 2022
Module retail
Fix
__KRING_DECAY_GRID
was aggregating the value of the moving window pivoting index instead of neighbour indexes.
November 8, 2022
Module h3
Improvement
Add linestrings and points support to function H3_POLYFILL.
November 7, 2022
Module retail
New
Add
FIND_TWIN_AREAS_WEIGHTED
procedure.
November 1, 2022
All modules
New
Create release workflows
October 28, 2022
Module s2
New
Add S2_RESOLUTION function.
Add S2_TOCHILDREN function.
October 26, 2022
Module transformations
Fix
Fix ST_BUFFER crashing with geographies close to the poles.
Module tiler
Improvement
Partition tables in spatial index tilesets.
Module cpg
New
Add
CUSTOMER_SEGMENTATION_ANALYSIS_DATA
procedure.Add
GENERATE_TRADE_AREAS
procedure.Add
RUN_CUSTOMER_SEGMENTATION
procedure.
Module lds
Fix
CREATE_ISOLINES output is stored in the column
geom
instead of__iso_geom
.
Improvement
CREATE_ISOLINES output is clustered by geometry.
October 24, 2022
Module processing
Fix
Prevent error in ST_VORONOIPOLYGONS, ST_VORONOILINES, ST_VORONOIPOLYGONS, ST_DELAUNAYLINES when points where too close together by rounding input coordinates to 5 decimal places.
October 7, 2022
Module clustering
New
Move ST_CLUSTERKMEANS function to core.
Module random
New
Move ST_GENERATEPOINTS function to core.
September 30, 2022
Module lds
Improvement
Rename
LDS_API_URL
toAPI_BASE_URL
.
September 29, 2022
Module retail
Improvement
LINEAR_REG
model is now supported.
September 27, 2022
Module tiler
Improvement
Fix bug in
__BUILD_REVENUE_MODEL_DATA_GRID
: Output table now containscompetitor_count
.
September 19, 2022
Module tiler
Improvement
Set
version
field in tilers metadata compliant with AT version.
September 17, 2022
Module tiler
Improvement
Hotfix: set
generate_feature_id
to false by default due a bug in the generation.
September 16, 2022
Module data
Improvement
Optimize DATAOBS_ENRICH_GRID, DATAOBS_ENRICH_GRID_RAW when the enrich data is a grid compatible with the input.
Remove support for s2, geohas and quadbinh in all enrich procedures.
Optimize all enrichment result tables by default by clusterizing by the geography or spatial index column.
September 15, 2022
Module s2
New
Add S2_CENTER function.
Module retail
Improvement
Return shap_values in PREDICT_REVENUE_AVERAGE.
August 26, 2022
Module tiler
New
Add
generate_feature_id
parameter inCREATE_SIMPLE_TILESET
.
August 19, 2022
Module retail
Improvement
Index parameter admits any type in PREDICT_REVENUE_AVERAGE.
Index from revenue_model_data is not casted in PREDICT_REVENUE_AVERAGE to reduce query data processing.
August 16, 2022
Module tiler
Fix
Fixed bug when parsing zoom columns in geography tilers.
August 11, 2022
Module tiler
Improvement
Unify
extra_metadata
intometadata
in tiler metadata.
August 9, 2022
Module h3
Fix
Apply make_valid in H3_BOUNDARY.
August 8, 2022
Module tiler
Fix
Fixed bug in metadata
CREATE_SPATIAL_INDEX_TILESET
.
August 5, 2022
Module retail
Improvement
Check fixed model options in BUILD_REVENUE_MODEL procedure.
August 3, 2022
Module retail
New
Add BUILD_CANNIBALIZATION_DATA procedure.
Add CANNIBALIZATION_OVERLAP procedure.
July 26, 2022
Module lds
New
Add GEOCODE_TABLE procedure.
Add LDS_QUOTA_INFO procedure.
Add GEOCODE_REVERSE_TABLE procedure.
Add CREATE_ISOLINES procedure.
July 21, 2022
Module statistics
Improvement
Added HH, LL, LH, HL detection in LOCAL_MORANS_I functions.
July 20, 2022
Module tiler
Fix
Fixed bug when passing multiline input to
CREATE_SPATIAL_INDEX_TILESET
.
July 8, 2022
Module data
New
Add quadbin support to module.
Module retail
New
Add quadbin support
Module tiler
New
Support of quadgrid as alias in
CREATE_POINT_AGGREGATION_TILESET
.
Improvement
Use core functions to convert H3 from int to string.
July 7, 2022
Module h3
Fix
Correctly handle large polygons in H3_POLYFILL.
Fixed wrong uppercase for quadbin and h3 tile ids
Module statistics
New
Add SMOOTHING_MRF_QUADBIN procedure.
Add LOCAL_MORANS_I_QUADBIN function.
Add MORANS_I_QUADBIN function.
Add GETIS_ORD_QUADBIN function.
Quadbin support in GWR_GRID function.
Module tiler
Fix
Fixed tablename quoting in BigQuery tiler
July 6, 2022
Module retail
Fix
Fix quadkey support in FIND_WHITESPACE_AREAS
June 29, 2022
Module retail
Fix
Fixing bugs when not providing store variables to BUILD_REVENUE_MODEL_DATA
June 23, 2022
Module quadbin
New
Add QUADBIN_BBOX function.
Add QUADBIN_BOUNDARY function.
Add QUADBIN_CENTER function.
Add QUADBIN_FROMGEOGPOINT function.
Add QUADBIN_FROMLONGLAT function.
Add QUADBIN_FROMZXY function.
Add QUADBIN_ISVALID function.
Add QUADBIN_KRING function.
Add QUADBIN_KRING_DISTANCES function.
Add QUADBIN_POLYFILL function.
Add QUADBIN_RESOLUTION function.
Add QUADBIN_SIBLING function.
Add QUADBIN_TOCHILDREN function.
Add QUADBIN_TOPARENT function.
Add QUADBIN_TOZXY function.
June 21, 2022
Module data
Fix
Fix BQ inplace enrichment with uppercase column names.
Module tiler
Improvement
Support H3 integer in
CREATE_SPATIAL_INDEX_TILESET
.
June 13, 2022
Module data
Fix
Fix order of
__KRING_DECAY
parameters.
June 10, 2022
Module tiler
Fix
Fix global stats in
CREATE_SPATIAL_INDEX_TILESET
.
Improvement
Check input resolution limits in
CREATE_SPATIAL_INDEX_TILESET
.
June 9, 2022
Module data
Fix
Report missing subscription geographies in DATAOBS_ENRICH_ procedures.
June 6, 2022
Module data
Fix
Fix stability issues in GRIDIFY_ENRICH procedure.
June 3, 2022
Module retail
New
Add store features and competitors stores features into BUILD_REVENUE_MODEL_DATA procedure.
Fix
Fixing bugs calling other routines
June 2, 2022
Module tiler
New
Add spatial index tileset generator
CREATE_SPATIAL_INDEX_TILESET
.
May 16, 2022
Module tiler
Fix
Bug fixing when the geometry early collapse
April 27, 2022
Module data
New
Add GRIDIFY_ENRICH procedure.
Module retail
New
Add FIND_TWIN_AREAS procedure.
April 22, 2022
Module data
New
Allow using tables as input, not just queries.
Add in place enrichment.
Allow using tables as input.
Accept NULL as output.
April 20, 2022
Module retail
Fix
Fix BUILD_REVENUE_MODEL_DATA k-ring buffer computation.
March 21, 2022
Module transformations
Improvement
ST_CONCAVEHULL now allows arrays with one/two points as input.
Module retail
New
Add COMMERCIAL_HOTSPOTS procedure.
Module statistics
New
Add SMOOTHING_MRF_H3 procedure.
Add SMOOTHING_MRF_QUADKEY procedure.
February 16, 2022
Module statistics
New
Add VARIOGRAM function.
Add ORDINARY_KRIGING function.
Add ORDINARY_KRIGING_TABLE procedure.
February 15, 2022
Module h3
New
Add H3_CENTER function.
Add H3_RESOLUTION function.
Module retail
New
Add FIND_WHITESPACE_AREAS procedure.
February 3, 2022
Module retail
New
Remove features importance tables from BUILD_REVENUE_MODEL procedure.
February 1, 2022
Module statistics
New
Add LOCAL_MORANS_I_H3 function.
Add LOCAL_MORANS_I_QUADKEY function.
Improvement
Improve MORANS_I_H3 and MORANS_I_QUADKEY implementation.
January 10, 2022
Module statistics
Fix
Replace STDDEV by STDDEV_POP in GETIS_ORD_H3 and GETIS_ORD_QUADKEY.
December 22, 2021
Module retail
New
Add BUILD_REVENUE_MODEL_DATA procedure.
Add BUILD_REVENUE_MODEL procedure.
Add PREDICT_REVENUE_AVERAGE procedure.
December 16, 2021
Module accessors
Improvement
Deployment schema "carto" instead of "accessors".
Removed
Remove VERSION function.
Module constructors
Improvement
Deployment schema "carto" instead of "constructors".
Removed
Remove VERSION function.
Module geohash
Improvement
Deployment schema "carto" instead of "geohash".
Removed
Remove VERSION function.
Module h3
Improvement
Deployment schema "carto" instead of "h3".
Rename ST_ASH3 function to H3_FROMGEOGPOINT.
Rename LONGLAT_ASH3 function to H3_FROMLONGLAT.
Rename ST_ASH3_POLYFILL function to H3_POLYFILL.
Rename ST_BOUNDARY function to H3_BOUNDARY.
Rename ISVALID function to H3_ISVALID.
Rename COMPACT function to H3_COMPACT.
Rename UNCOMPACT function to H3_UNCOMPACT.
Rename TOPARENT function to H3_TOPARENT.
Rename TOCHILDREN function to H3_TOCHILDREN.
Rename ISPENTAGON function to H3_ISPENTAGON.
Rename DISTANCE function to H3_DISTANCE.
Rename KRING function to H3_KRING.
Rename KRING_DISTANCES function to H3_KRING_DISTANCES.
Rename HEXRING function to H3_HEXRING.
Removed
Remove VERSION function.
Module measurements
Improvement
Deployment schema "carto" instead of "measurements".
Removed
Remove ST_ANGLE, already present in Bigquery.
Remove VERSION function.
Module placekey
Improvement
Deployment schema "carto" instead of "placekey".
Rename H3_ASPLACEKEY function to PLACEKEY_FROMH3.
Rename PLACEKEY_ASH3 function to PLACEKEY_TOH3.
Rename ISVALID function to PLACEKEY_ISVALID.
Removed
Remove VERSION function.
Module processing
Improvement
Deployment schema "carto" instead of "processing".
Removed
Remove VERSION function.
Module s2
Improvement
Deployment schema "carto" instead of "s2".
Rename ID_FROMHILBERTQUADKEY function to S2_FROMHILBERTQUADKEY.
Rename HILBERTQUADKEY_FROMID function to S2_TOHILBERTQUADKEY.
Rename LONGLAT_ASID function to S2_FROMLONGLAT.
Rename ST_ASID function to S2_FROMGEOGPOINT.
Rename ID_FROMTOKEN function to S2_FROMTOKEN.
Rename TOKEN_FROMID function to S2_TOTOKEN.
Rename ID_FROMUINT64REPR function to S2_FROMUINT64REPR.
Rename UINT64REPR_FROMID function to S2_TOUINT64REPR.
Rename ST_BOUNDARY function to S2_BOUNDARY.
Removed
Remove VERSION function.
Module transformations
Improvement
Deployment schema "carto" instead of "transformations".
Removed
Remove VERSION function.
Module clustering
Improvement
Deployment schema "carto" instead of "clustering".
Removed
Remove VERSION function.
Module data
Improvement
Deployment schema "carto" instead of "data".
Removed
Remove VERSION function.
Module random
Improvement
Deployment schema "carto" instead of "random".
Removed
Remove VERSION function.
Module routing
Improvement
Deployment schema "carto" instead of "routing".
Removed
Remove VERSION function.
Module statistics
Improvement
Deployment schema "carto" instead of "statistics".
Removed
Remove VERSION function.
Module tiler
Improvement
Deployment schema "carto" instead of "tiler".
Removed
Remove VERSION function.
December 13, 2021
Module tiler
Improvement
Add support for legacy project names that contain dots or other special characters.
December 10, 2021
Module statistics
New
Add KNN function.
Add GFUN function.
Add LOF function.
Add LOF_TABLE procedure.
November 30, 2021
Module tiler
Fix
Escape single-quoted properties in the
CREATE_TILESET
metadata when obtaining the geometry type.
November 25, 2021
Module statistics
New
Add GWR_GRID function.
November 16, 2021
Module tiler
Fix
Use
zoom_min_column
aszoom_min
andzoom_max_column
aszoom_max
when provided.
Improvement
Optimize simplification method instead of the native BigQuery.
November 5, 2021
Module data
Improvement
Fields named
dimension
,total
,intersection
andinput_area
are now__carto_dimension
,__carto_total
and__carto_intersection
and__carto_input_area
. Also the column_carto_enrichment_
is now__carto_enrichment
. This affects all the raw enrichment procedures:ENRICH_POINTS_RAW
,ENRICH_POLYGONS_RAW
,ENRICH_GRID_RAW
,DATAOBS_ENRICH_POINTS_RAW
,DATAOBS_ENRICH_POLYGONS_RAW
,DATAOBS_ENRICH_GRID_RAW
.
Fix
User provided queries can now have columns named
dimension
,total
,intersection
,input_area
,_nonglobal
, which could have collided previously with internal columns. All internal columns are now prefixed with__carto_
. This affects all the enrichment procedures:ENRICH_POINTS
,ENRICH_POLYGONS
,ENRICH_GRID
,DATAOBS_ENRICH_POINTS
,DATAOBS_ENRICH_POLYGONS
,DATAOBS_ENRICH_GRID
,ENRICH_POINTS_RAW
,ENRICH_POLYGONS_RAW
,ENRICH_GRID_RAW
,DATAOBS_ENRICH_POINTS_RAW
,DATAOBS_ENRICH_POLYGONS_RAW
,DATAOBS_ENRICH_GRID_RAW
.
November 2, 2021
Module data
Improvement
Add
associated_geography_slug
column toDATAOBS_SUBSCRIPTIONS
.Some column names have changed in
DATAOBS_SUBSCRIPTIONS
:table
is nowdataset_table
associated_geography
is nowassociated_geography_table
Some column names have changed in
DATAOBS_SUBSCRIPTION_VARIABLES
:geography_slug
is nowassociated_geography_slug
.
Some column names have changed in
DATAOBS_SAMPLES
:table
is nowdataset_sample_table
.
In
DATAOBS_SUBSCRIPTION_VARIABLES
, the columnassociated_geography_slug
is now NULL for geography datasets.
Fix
The output of
DATAOBS_SUBSCRIPTIONS
andDATAOBS_SUBSCRIPTION_VARIABLES
now omits items that are not Data Observatory products.
October 15, 2021
Module data
Fix
Enrichment results could mix input rows and values, specially for large inputs due to multiple evaluation of nondeterministic ROW_NUMBERS expression.
Improvement
New API, with consistent naming of procedures `
September 23, 2021
Module s2
Improvement
Rename functions ID_FROMUINT64REPR, UINT64REPR_FROMID to follow convention.
Module statistics
New
Create statistics module.
Add GETIS_ORD_H3 function.
Add GETIS_ORD_QUADKEY function.
Add MORANS_I_H3 function.
Add MORANS_I_QUADKEY function.
Add VERSION function.
September 22, 2021
Module h3
Improvement
Review HEXRING, KRING functions.
Change KRING_INDEXED to KRING_DISTANCES.
September 17, 2021
Module tiler
Fix
Fix support for DATE in
CREATE_TILESET
.Fix naming of internal variables to avoid name collisions.
New
Add
fraction_dropped_per_zoom
in tileset metadata.
September 16, 2021
Module data
Changes
Support
sub_
(table) subscriptions in:DATAOBS_ENRICH_POINTS
DATAOBS_ENRICH_POINTS_WITH_MEASURES
DATAOBS_ENRICH_POLYGONS_WITH_AGGREGATION
DATAOBS_ENRICH_POLYGONS_WITH_MEASURES
DATAOBS_ENRICH_GRID
September 14, 2021
Module s2
Fix
Avoid keeping planar shape in spherical coordinates in ST_BOUNDARY.
September 3, 2021
Module data
Changes
Search for tables in DATAOBS_SAMPLES, DATAOBS_SUBSCRIPTIONS, DATAOBS_SUBSCRIPTIONS_VARIABLES.
August 31, 2021
Module data
Improvement
DATAOBS_ENRICH_* procedures are now optimized to reduce the amount of data processed by taking advantage of clusterization by geography in the Data Observatory.
The
input_id_column
parameter has been removed from the following procedures:DATAOBS_ENRICH_POINTS
DATAOBS_ENRICH_POINTS_WITH_MEASURES
DATAOBS_ENRICH_POLYGONS_WITH_AGGREGATION
DATAOBS_ENRICH_POLYGONS_WITH_MEASURES
ENRICH_POINTS
ENRICH_POINTS_WITH_MEASURES
ENRICH_POLYGONS_WITH_AGGREGATION
ENRICH_POLYGONS_WITH_MEASURES
August 26, 2021
Module data
Improvement
ENRICH_GRID and DATAOBS_ENRICH_GRID procedures take
input_query
andinput_index_column
parameters instead of theindices
array.DATAOBS_ENRICH_* procedures now report missing subscriptions in a more user-friendly way.
Module routing
Improvement
Pass network to JS UDF natively
Support custom speed over LineStrings for GENERATE_NETWORK function.
Support custom speed over LineStrings for FIND_SHORTEST_PATH function.
Support custom speed over LineStrings for DISTANCE_MAP function.
August 25, 2021
Module tiler
Fix
Fix support for TIMESTAMP,DATETIME in CREATE_TILESET and CREATE_SIMPLE_TILESET.
Reduce parallelization to skip the recent error: "Not enough resources for query planning - too many subqueries or query is too complex"
August 24, 2021
Module h3
Fix
Support GEOMETRYCOLLECTION from ST_ASH3_POLYFILL.
August 6, 2021
Module data
New
Add DATAOBS_ENRICH_GRID procedure.
Add ENRICH_GRID procedure.
August 4, 2021
Module h3
New
Add KRING_INDEXED function.
July 30, 2021
Module geohash
New
Create geohash module.
Add VERSION function.
Add ST_BOUNDARY function.
Module tiler
Improvement
Checking if the output table exists before running the tiler.
July 13, 2021
Module data
Improvement
DATAOBS_ENRICH_* procedures take single
source
parameter with defaultcarto-customers
project.
July 9, 2021
Module data
New
Add DATAOBS_SAMPLES procedure.
Improvement
DATAOBS_SUBSCRIPTIONS and DATAOBS_SUBSCRIPTION_VARIABLES take single
source
parameter with defaultcarto-customers
project. Views withsub_
prefix are supported.
Module routing
New
Create routing module.
Add GENERATE_NETWORK function.
Add GENERATE_NETWORK_TABLE procedure.
Add FIND_SHORTEST_PATH function.
Add FIND_SHORTEST_PATH_FROM_NETWORK function.
Add FIND_SHORTEST_PATH_FROM_NETWORK_TABLE procedure.
Add DISTANCE_MAP function.
Add DISTANCE_MAP_FROM_NETWORK function.
Add DISTANCE_MAP_FROM_NETWORK_TABLE procedure.
Add VERSION function.
July 8, 2021
Module data
Fix
Fix DATAOBS_SUBSCRIPTIONS and DATAOBS_SUBSCRIPTION_VARIABLES not working with subscriptions in which the provider id contained underscores.
July 1, 2021
Module tiler
New
Add new automatic tileset generator
CREATE_TILESET
.
June 23, 2021
Module data
New
Add DATAOBS_SUBSCRIPTIONS procedure.
Add DATAOBS_SUBSCRIPTION_VARIABLES procedure.
Improvement
Change spatial catalog structure using two tables.
Improve enrichment procedures:
Use public geographies instead of subscription views
Remove unnecessary variables from ENRICH_POLYGONS_WITH_MEASURES
Make input_id_column always mandatory
Add input_area consistently
June 2, 2021
Module data
New
Create data module.
Add VERSION function.
Add DATAOBS_ENRICH_POINTS procedure.
Add DATAOBS_ENRICH_POINTS_WITH_MEASURES procedure.
Add DATAOBS_ENRICH_POLYGONS_WITH_AGGREGATION procedure.
Add DATAOBS_ENRICH_POLYGONS_WITH_MEASURES procedure.
Add ENRICH_POINTS procedure.
Add ENRICH_POINTS_WITH_MEASURES procedure.
Add ENRICH_POLYGONS_WITH_AGGREGATION procedure.
Add ENRICH_POLYGONS_WITH_MEASURES procedure.
June 1, 2021
Module s2
New
Add TOKEN_FROMID function.
Add ID_FROMTOKEN function.
Add ID_FROM_UINT64REPR function.
Add UINT64REPR_FROM_ID function.
May 19, 2021
Module tiler
Improvement
Use semantic versioning.
Add
features_count
andgeometry
labels to the tilesets.
May 6, 2021
Module clustering
Improvement
Change ST_CLUSTERKMEANS returning cluster and geom
May 4, 2021
Module accessors
New
Create accessors module.
Add ST_ENVELOPE function.
Add VERSION function.
Module processing
New
Create processing module.
Add ST_VORONOIPOLYGONS function.
Add ST_VORONOILINES function.
Add ST_DELAUNAYPOLYGONS function.
Add ST_DELAUNAYLINES function.
Add ST_POLYGONIZE function.
Add VERSION function.
Module transformations
New
Add ST_CONCAVEHULL function.
April 30, 2021
Module clustering
New
Create clustering module.
Add ST_CLUSTERKMEANS function.
Add VERSION function.
April 29, 2021
Module constructors
New
Add ST_BEZIERSPLINE function.
Add ST_MAKEELLIPSE function.
Module measurements
New
Create measurements module.
Add ST_ANGLE function.
Add ST_AZIMUTH function.
Add ST_MINKOWSKIDISTANCE function.
Module transformations
New
Rename module to transformations.
Add ST_CENTERMEAN function.
Add ST_CENTERMEDIAN function.
Add ST_CENTEROFMASS function.
Add ST_DESTINATION function.
Add ST_GREATCIRCLE function.
Add ST_LINE_INTERPOLATE_POINT function.
Module tiler
Removed
Remove quota & project checking.
April 28, 2021
Module constructors
New
Create constructors module.
Add ST_MAKEENVELOPE function.
Add ST_TILEENVELOPE function.
Add VERSION function.
April 16, 2021
Module transformations
New
Create transformation module.
Add ST_BUFFER function.
Add VERSION function.
April 10, 2021
Module random
Improvement
ST_GENERATEPOINTS now uses a spherically uniform distribution. Previously used to by uniform on projection.
April 9, 2021
Module h3
Improvement
Use hexadecimal as default type instead of int for h3 indexes.
Fix
Fix ST_BOUNDARY generating error when not able to parse geometry.
Module placekey
Improvement
Placekey conversions works with hexadecimal h3 indexes instead of int.
Module random
New
Create random module.
Add ST_GENERATEPOINTS function.
Add VERSION function.
March 31, 2021
Module h3
New
Create h3 module.
Add ST_ASH3 function.
Add LONGLAT_ASH3 function.
Add ST_ASH3_POLYFILL function.
Add ST_BOUNDARY function.
Add ISVALID function.
Add COMPACT function.
Add UNCOMPACT function.
Add TOPARENT function.
Add TOCHILDREN function.
Add ISPENTAGON function.
Add DISTANCE function.
Add KRING function.
Add HEXRING function.
Add VERSION function.
Module placekey
New
Create placekey module.
Add H3_ASPLACEKEY function.
Add PLACEKEY_ASH3 function.
Add ISVALID function.
Add VERSION function.
Module s2
New
Create s2 module.
Add ID_FROMHILBERTQUADKEY function.
Add HILBERTQUADKEY_FROMID function.
Add LONGLAT_ASID function.
Add ST_ASID function.
Add ST_BOUNDARY function.
Add VERSION function.
Module tiler
New
Add QUOTA procedure in order to display the monthly quota and quota available.
Implement support for layer name.
Add a max limit to the
max_tile_size_kb
property. Adds a new option (skip_validation
) to circunvent this limit, which also allows going over the max partition limit.Add a limit to the max amount of expected tiles at the highest zoom level for Simple Tilesets. It can be disabled via the
skip_validation
option of modified (for tests usingmax_tiles_at_maxzoom
).Add new
drop_fraction_as_needed
strategy initial implementation. Also adds internaldebug
andfraction_drop_per_zoom_level
options to help debugging.Add new
metadada
input parameter to set the propertiesname
,description
,legends
and alsoextra_metadata
.Add
zoom_min_column
andzoom_max_column
to control the zoom level at which particular features are visible.
Improvement
Change the default and max partition limit to 3999.
Make aggregation tilesets to use the quadkey module UDF LONGLAT_ASQUADINTLIST_RESOLUTION in order to generate quadkey aggregations.
Removed
Removed
zoom_step
option.
Fix
Fix bug that left empty the generator options in the metadata cell.
Fix CREATE_SIMPLE_TILESET failing with single column tables.
Fix collision with
frequency
column name.
February 10, 2021
Module tiler
New
Add tileset type to metadata and BigQuery labels.
Add the prefix "aggregation_" to aggregation options: type, resolution and placement.
Improvement
Moved to the Advanced Spatial Extension.
Change "max_tile_size_strategy" default to "throw_error", matching the other defaults and improve the exception error message in CreatePointAggregationTileset.
Rename adapting to BigQuery conventions, using UPPER CASE and separating words with underscore "_".
Set up limits for the procedures based on a limits table in the config project. The users can only execute the tiler from the "billing_project" and they have a maximum amount of processed bytes per month.
Removed
Remove compression levels 10-12 to reduce library size.
Remove ST prefix from functions not using geographies.
Last updated