Links

Release notes

May 5, 2023 (v 1.3.0)

Module statistics
New
  • Set of procedures to compute Area of Applicability of a model built with BigQuery ML:
    • AREA_OF_APPLICABILITY: This procedure computes the Area of Applicability (AOA) of a Bigquery ML model. It generates a metric which tells the user where the results from a Machine Learning (ML) model can be trusted when the predictions are extrapolated outside the training space (i.e. where the estimated cross-validation performance holds).
    • ENV_BLOCKING: This procedure derives cross validation (CV) folds based on environmental blocking.
Fix
  • We have added a new mandatory parameter called correlation_var when scoring_method = FIRST_PC in the CREATE_SPATIAL_COMPOSITE_UNSUPERVISED
  • We have improved the management of temporary tables in all procedures to compute spatial composites (scores)
Module raster (beta)
Improvement

April 4, 2023 (v 1.2.0)

Module statistics
New
  • We have released a new set of procedures within the statistics module to enable users to create spatial scores (also known as composite indicators or indexes) derived from a combination of different features. We have included 3 different procedures:
Module lds
Change
  • We have updated the LDS module to be aligned with the new usage quota model, working as an annual budget and combining consumption across all supported LDS services.
Fix
  • GEOCODE_TABLE now supports null values in the input address column, ignoring them when running the geocoding process to provide results only for the informed addresses.

March 7, 2023 (v 1.1.0)

Module raster (beta)
Improvement
Note
The raster module of the Analytics Toolbox for BigQuery v1.1.0 will work only with raster-loader v0.3.0, which pre-computes and stores metadata in the raster table and homogenizes the generic/quadbin raster table format. So, if you update your Analytics Toolbox version to the most recent v1.1.0, please ensure you are using the raster-loader v0.3.0.
  • We have made the RASTER_GETMETADATA faster by retrieving metadata from the table and avoiding creating again the original table.
  • We have improved the RASTER_ST_GETVALUE by removing pixel_geog column and cluster results by quadbin in the quadbin raster format.
  • We have enhanced the RASTER_ST_GETVALUE_FROM_TABLE outcome by preserving the input columns, renaming centroid_quadbin_block and centroid_quadbin_pixel to block_quadbin and quadbin, clustering the results by quadbin in quadbin raster and removing few non-informative columns (e.g. geog and myoffset).
Module cpg
New
  • We have released a new set of functions to enable users to perform the merchant universe matching analysis. It includes 2 different functions, UNIVERSE_MATCHING which performs a fuzzy match between two different POI datasets based on location and name similarity (i.e. one dataset should include current merchants and the other one the complete universe of all potential merchants), and UNIVERSE_MATCHING_REPORT that generates report-like tables summarizing market penetration based on the results of the UNIVERSE_MATCHING analysis.
Fix
Module lds
Improvement
  • From this version, we support line break (\n) and other escaped sequences in addresses in GEOCODE_TABLE.

January 31, 2023 (v 1.0.0)

Raster module (beta)
New
Note that Analytics Toolbox for BigQuery v1.0.0 will work only with raster-loader v0.2.0, to use the latest version of rater-loader please ensure you also have access to the latest version of the Analytics Toolbox
Routing module (beta)
Improvement
  • New enhanced version of the routing module with procedures ROUTING_MATRIX to calculate origin-destination matrices and ROUTING_ISOLINES to compute isolines around a set of locations, both supporting multiple transportation modes (car, bike, and walk). These new functions run on top of CARTO’s road network (derived from OSM segments) that is available as a public subscription in the Data Observatory. Please note that these improvements imply breaking changes with the previous version of the routing module.
Module retail
Fix
Module statistics
New
  • Add IDW function (Inverse Distance Weighted).
Module tiler
Improvement
  • Update default zoom values: geo, quadbin, h3 (12, 12, 6).
  • Fix spatial index tiler with empty properties.

December 22, 2022

Module lds
Improvement
  • We have added the possibility to configure more options as parameters when executing the functions to CREATE_ISOLINES. These new options, which depend on the LDS service provider, allow the user to configure more transportation modes such as truck or bike, the possibility of specifying departure or arrival times allowing the creation of reverse isolines, and other options like different routing modes. Please note that these improvements imply breaking changes with the previous version of the CREATE_ISOLINES functions.
  • Added new confidence/relevance metadata to the results of the geocoding function GEOCODE_TABLE. Please note that this improvement has implied breaking changes with the previous versions of the GEOCODE_TABLE and GEOCODE_REVERSE_TABLE functions.
Module cpg
New
  • New FIND_SIMILAR_LOCATIONS function to the CPG module, that allows users to identify which locations (e.g. merchants, stores) are more similar to a chosen location (e.g. top performant) based on the characteristics of their surrounding areas (or trade areas).
Module retail
Improvement
  • Update in BUILD_CANNIBALIZATION_DATA and CANNIBALIZATION_OVERLAP to support buffer, kring and isoline methods. Please note that these improvements have implied some breaking changes in the aforementioned functions invalidating their previous version.
Module data
Fix
  • We have implemented a fix in the weighted averages computation in the GRIDIFY_ENRICH procedure.
Module quadbin
New
  • Quadbin/quadkey conversion functions have been added to ease the conversion from quadbin to quadkey and from quadkey to quadbin indexes.

November 15, 2022

Module retail
Fix
  • __KRING_DECAY_GRID was aggregating the value of the moving window pivoting index instead of neighbour indexes.

November 8, 2022

Module h3
Improvement
  • Add linestrings and points support to function H3_POLYFILL.

November 7, 2022

Module retail
New
  • Add FIND_TWIN_AREAS_WEIGHTED procedure.

November 1, 2022

All modules
New
  • Create release workflows

October 28, 2022

Module s2
New

October 26, 2022

Module transformations
Fix
  • Fix ST_BUFFER crashing with geographies close to the poles.
Module tiler
Improvement
  • Partition tables in spatial index tilesets.
Module cpg
New
  • Add CUSTOMER_SEGMENTATION_ANALYSIS_DATA procedure.
  • Add GENERATE_TRADE_AREAS procedure.
  • Add RUN_CUSTOMER_SEGMENTATION procedure.
Module lds
Fix
  • CREATE_ISOLINES output is stored in the column geom instead of __iso_geom.
Improvement
  • CREATE_ISOLINES output is clustered by geometry.

October 24, 2022

Module processing
Fix
  • Prevent error in ST_VORONOIPOLYGONS, ST_VORONOILINES, ST_VORONOIPOLYGONS, ST_DELAUNAYLINES when points where too close together by rounding input coordinates to 5 decimal places.

October 7, 2022

Module clustering
New
  • Move ST_CLUSTERKMEANS function to core.
Module random
New
  • Move ST_GENERATEPOINTS function to core.

September 30, 2022

Module lds
Improvement
  • Rename LDS_API_URL to API_BASE_URL.

September 29, 2022

Module retail
Improvement
  • LINEAR_REG model is now supported.

September 27, 2022

Module tiler
Improvement
  • Fix bug in __BUILD_REVENUE_MODEL_DATA_GRID: Output table now contains competitor_count.

September 19, 2022

Module tiler
Improvement
  • Set version field in tilers metadata compliant with AT version.

September 17, 2022

Module tiler
Improvement
  • Hotfix: set generate_feature_id to false by default due a bug in the generation.

September 16, 2022

Module data
Improvement
  • Optimize DATAOBS_ENRICH_GRID, DATAOBS_ENRICH_GRID_RAW when the enrich data is a grid compatible with the input.
  • Remove support for s2, geohas and quadbinh in all enrich procedures.
  • Optimize all enrichment result tables by default by clusterizing by the geography or spatial index column.

September 15, 2022

Module s2
New
  • Add S2_CENTER function.
Module retail
Improvement
  • Return shap_values in PREDICT_REVENUE_AVERAGE.

August 26, 2022

Module tiler
New
  • Add generate_feature_id parameter in CREATE_SIMPLE_TILESET.

August 19, 2022

Module retail
Improvement
  • Index parameter admits any type in PREDICT_REVENUE_AVERAGE.
  • Index from revenue_model_data is not casted in PREDICT_REVENUE_AVERAGE to reduce query data processing.

August 16, 2022

Module tiler
Fix
  • Fixed bug when parsing zoom columns in geography tilers.

August 11, 2022

Module tiler
Improvement
  • Unify extra_metadata into metadata in tiler metadata.

August 9, 2022

Module h3
Fix
  • Apply make_valid in H3_BOUNDARY.

August 8, 2022

Module tiler
Fix
  • Fixed bug in metadata CREATE_SPATIAL_INDEX_TILESET.

August 5, 2022

Module retail
Improvement
  • Check fixed model options in BUILD_REVENUE_MODEL procedure.

August 3, 2022

Module retail
New
  • Add BUILD_CANNIBALIZATION_DATA procedure.
  • Add CANNIBALIZATION_OVERLAP procedure.

July 26, 2022

Module lds
New
  • Add GEOCODE_TABLE procedure.
  • Add LDS_QUOTA_INFO procedure.
  • Add GEOCODE_REVERSE_TABLE procedure.
  • Add CREATE_ISOLINES procedure.

July 21, 2022

Module statistics
Improvement
  • Added HH, LL, LH, HL detection in LOCAL_MORANS_I functions.

July 20, 2022

Module tiler
Fix
  • Fixed bug when passing multiline input to CREATE_SPATIAL_INDEX_TILESET.

July 8, 2022

Module data
New
  • Add quadbin support to module.
Module retail
New
  • Add quadbin support
Module tiler
New
  • Support of quadgrid as alias in CREATE_POINT_AGGREGATION_TILESET.
Improvement
  • Use core functions to convert H3 from int to string.

July 7, 2022

Module h3
Fix
  • Correctly handle large polygons in H3_POLYFILL.
  • Fixed wrong uppercase for quadbin and h3 tile ids
Module statistics
New
  • Add SMOOTHING_MRF_QUADBIN procedure.
  • Add LOCAL_MORANS_I_QUADBIN function.
  • Add MORANS_I_QUADBIN function.
  • Add GETIS_ORD_QUADBIN function.
  • Quadbin support in GWR_GRID function.
Module tiler
Fix
  • Fixed tablename quoting in BigQuery tiler

July 6, 2022

Module retail
Fix
  • Fix quadkey support in FIND_WHITESPACE_AREAS

June 29, 2022

Module retail
Fix
  • Fixing bugs when not providing store variables to BUILD_REVENUE_MODEL_DATA

June 23, 2022

Module quadbin
New
  • Add QUADBIN_BBOX function.
  • Add QUADBIN_BOUNDARY function.
  • Add QUADBIN_CENTER function.
  • Add QUADBIN_FROMGEOGPOINT function.
  • Add QUADBIN_FROMLONGLAT function.
  • Add QUADBIN_FROMZXY function.
  • Add QUADBIN_ISVALID function.
  • Add QUADBIN_KRING function.
  • Add QUADBIN_KRING_DISTANCES function.
  • Add QUADBIN_POLYFILL function.
  • Add QUADBIN_RESOLUTION function.
  • Add QUADBIN_SIBLING function.
  • Add QUADBIN_TOCHILDREN function.
  • Add QUADBIN_TOPARENT function.
  • Add QUADBIN_TOZXY function.

June 21, 2022

Module data
Fix
  • Fix BQ inplace enrichment with uppercase column names.
Module tiler
Improvement
  • Support H3 integer in CREATE_SPATIAL_INDEX_TILESET.

June 13, 2022

Module data
Fix
  • Fix order of __KRING_DECAY parameters.

June 10, 2022

Module tiler
Fix
  • Fix global stats in CREATE_SPATIAL_INDEX_TILESET.
Improvement
  • Check input resolution limits in CREATE_SPATIAL_INDEX_TILESET.

June 9, 2022

Module data
Fix
  • Report missing subscription geographies in DATAOBS_ENRICH_ procedures.

June 6, 2022

Module data
Fix
  • Fix stability issues in GRIDIFY_ENRICH procedure.

June 3, 2022

Module retail
New
  • Add store features and competitors stores features into BUILD_REVENUE_MODEL_DATA procedure.
Fix
  • Fixing bugs calling other routines

June 2, 2022

Module tiler
New
  • Add spatial index tileset generator CREATE_SPATIAL_INDEX_TILESET.

May 16, 2022

Module tiler
Fix
  • Bug fixing when the geometry early collapse

April 27, 2022

Module data
New
  • Add GRIDIFY_ENRICH procedure.
Module retail
New
  • Add FIND_TWIN_AREAS procedure.

April 22, 2022

Module data
New
  • Allow using tables as input, not just queries.
  • Add in place enrichment.
  • Allow using tables as input.
  • Accept NULL as output.

April 20, 2022

Module retail
Fix
  • Fix BUILD_REVENUE_MODEL_DATA k-ring buffer computation.

March 21, 2022

Module transformations
Improvement
  • ST_CONCAVEHULL now allows arrays with one/two points as input.
Module retail
New
  • Add COMMERCIAL_HOTSPOTS procedure.
Module statistics
New
  • Add SMOOTHING_MRF_H3 procedure.
  • Add SMOOTHING_MRF_QUADKEY procedure.

February 16, 2022

Module statistics
New
  • Add VARIOGRAM function.
  • Add ORDINARY_KRIGING function.
  • Add ORDINARY_KRIGING_TABLE procedure.

February 15, 2022

Module h3
New
  • Add H3_CENTER function.
  • Add H3_RESOLUTION function.
Module retail
New
  • Add FIND_WHITESPACE_AREAS procedure.

February 3, 2022

Module retail
New
  • Remove features importance tables from BUILD_REVENUE_MODEL procedure.

February 1, 2022

Module statistics
New
  • Add LOCAL_MORANS_I_H3 function.
  • Add LOCAL_MORANS_I_QUADKEY function.
Improvement
  • Improve MORANS_I_H3 and MORANS_I_QUADKEY implementation.

January 10, 2022

Module statistics
Fix
  • Replace STDDEV by STDDEV_POP in GETIS_ORD_H3 and GETIS_ORD_QUADKEY.

December 22, 2021

Module retail
New
  • Add BUILD_REVENUE_MODEL_DATA procedure.
  • Add BUILD_REVENUE_MODEL procedure.
  • Add PREDICT_REVENUE_AVERAGE procedure.

December 16, 2021

Module accessors
Improvement
  • Deployment schema "carto" instead of "accessors".
Removed
  • Remove VERSION function.
Module constructors
Improvement
  • Deployment schema "carto" instead of "constructors".
Removed
  • Remove VERSION function.
Module geohash
Improvement
  • Deployment schema "carto" instead of "geohash".
Removed
  • Remove VERSION function.
Module h3
Improvement
  • Deployment schema "carto" instead of "h3".
  • Rename ST_ASH3 function to H3_FROMGEOGPOINT.
  • Rename LONGLAT_ASH3 function to H3_FROMLONGLAT.
  • Rename ST_ASH3_POLYFILL function to H3_POLYFILL.
  • Rename ST_BOUNDARY function to H3_BOUNDARY.
  • Rename ISVALID function to H3_ISVALID.
  • Rename COMPACT function to H3_COMPACT.
  • Rename UNCOMPACT function to H3_UNCOMPACT.
  • Rename TOPARENT function to H3_TOPARENT.
  • Rename TOCHILDREN function to H3_TOCHILDREN.
  • Rename ISPENTAGON function to H3_ISPENTAGON.
  • Rename DISTANCE function to H3_DISTANCE.
  • Rename KRING function to H3_KRING.
  • Rename KRING_DISTANCES function to H3_KRING_DISTANCES.
  • Rename HEXRING function to H3_HEXRING.
Removed
  • Remove VERSION function.
Module measurements
Improvement
  • Deployment schema "carto" instead of "measurements".
Removed
  • Remove ST_ANGLE, already present in Bigquery.
  • Remove VERSION function.
Module placekey
Improvement
  • Deployment schema "carto" instead of "placekey".
  • Rename H3_ASPLACEKEY function to PLACEKEY_FROMH3.
  • Rename PLACEKEY_ASH3 function to PLACEKEY_TOH3.
  • Rename ISVALID function to PLACEKEY_ISVALID.
Removed
  • Remove VERSION function.
Module processing
Improvement
  • Deployment schema "carto" instead of "processing".
Removed
  • Remove VERSION function.
Module s2
Improvement
  • Deployment schema "carto" instead of "s2".
  • Rename ID_FROMHILBERTQUADKEY function to S2_FROMHILBERTQUADKEY.
  • Rename HILBERTQUADKEY_FROMID function to S2_TOHILBERTQUADKEY.
  • Rename LONGLAT_ASID function to S2_FROMLONGLAT.
  • Rename ST_ASID function to S2_FROMGEOGPOINT.
  • Rename ID_FROMTOKEN function to S2_FROMTOKEN.
  • Rename TOKEN_FROMID function to S2_TOTOKEN.
  • Rename ID_FROMUINT64REPR function to S2_FROMUINT64REPR.
  • Rename UINT64REPR_FROMID function to S2_TOUINT64REPR.
  • Rename ST_BOUNDARY function to S2_BOUNDARY.
Removed
  • Remove VERSION function.
Module transformations
Improvement
  • Deployment schema "carto" instead of "transformations".
Removed
  • Remove VERSION function.
Module clustering
Improvement
  • Deployment schema "carto" instead of "clustering".
Removed
  • Remove VERSION function.
Module data
Improvement
  • Deployment schema "carto" instead of "data".
Removed
  • Remove VERSION function.
Module random
Improvement
  • Deployment schema "carto" instead of "random".
Removed
  • Remove VERSION function.
Module routing
Improvement
  • Deployment schema "carto" instead of "routing".
Removed
  • Remove VERSION function.
Module statistics
Improvement
  • Deployment schema "carto" instead of "statistics".
Removed
  • Remove VERSION function.
Module tiler
Improvement
  • Deployment schema "carto" instead of "tiler".
Removed
  • Remove VERSION function.

December 13, 2021

Module tiler
Improvement
  • Add support for legacy project names that contain dots or other special characters.

December 10, 2021

Module statistics
New
  • Add KNN function.
  • Add GFUN function.
  • Add LOF function.
  • Add LOF_TABLE procedure.

November 30, 2021

Module tiler
Fix
  • Escape single-quoted properties in the CREATE_TILESET metadata when obtaining the geometry type.

November 25, 2021

Module statistics
New
  • Add GWR_GRID function.

November 16, 2021

Module tiler
Fix
  • Use zoom_min_column as zoom_min and zoom_max_column as zoom_max when provided.
Improvement
  • Optimize simplification method instead of the native BigQuery.

November 5, 2021

Module data
Improvement
  • Fields named dimension, total, intersection and input_area are now __carto_dimension, __carto_total and __carto_intersection and __carto_input_area. Also the column _carto_enrichment_ is now __carto_enrichment. This affects all the raw enrichment procedures: ENRICH_POINTS_RAW, ENRICH_POLYGONS_RAW, ENRICH_GRID_RAW, DATAOBS_ENRICH_POINTS_RAW, DATAOBS_ENRICH_POLYGONS_RAW, DATAOBS_ENRICH_GRID_RAW.
Fix
  • User provided queries can now have columns named dimension, total, intersection, input_area, _nonglobal, which could have collided previously with internal columns. All internal columns are now prefixed with __carto_. This affects all the enrichment procedures: ENRICH_POINTS, ENRICH_POLYGONS, ENRICH_GRID, DATAOBS_ENRICH_POINTS, DATAOBS_ENRICH_POLYGONS, DATAOBS_ENRICH_GRID, ENRICH_POINTS_RAW, ENRICH_POLYGONS_RAW, ENRICH_GRID_RAW, DATAOBS_ENRICH_POINTS_RAW, DATAOBS_ENRICH_POLYGONS_RAW, DATAOBS_ENRICH_GRID_RAW.

November 2, 2021

Module data
Improvement
  • Add associated_geography_slug column to DATAOBS_SUBSCRIPTIONS.
  • Some column names have changed in DATAOBS_SUBSCRIPTIONS:
    • table is now dataset_table
    • associated_geography is now associated_geography_table
  • Some column names have changed in DATAOBS_SUBSCRIPTION_VARIABLES:
    • geography_slug is now associated_geography_slug.
  • Some column names have changed in DATAOBS_SAMPLES:
    • table is now dataset_sample_table.
  • In DATAOBS_SUBSCRIPTION_VARIABLES, the column associated_geography_slug is now NULL for geography datasets.
Fix
  • The output of DATAOBS_SUBSCRIPTIONS and DATAOBS_SUBSCRIPTION_VARIABLES now omits items that are not Data Observatory products.

October 15, 2021

Module data
Fix
  • Enrichment results could mix input rows and values, specially for large inputs due to multiple evaluation of nondeterministic ROW_NUMBERS expression.
Improvement
  • New API, with consistent naming of procedures `

September 23, 2021

Module s2
Improvement
  • Rename functions ID_FROMUINT64REPR, UINT64REPR_FROMID to follow convention.
Module statistics
New
  • Create statistics module.
  • Add GETIS_ORD_H3 function.
  • Add GETIS_ORD_QUADKEY function.
  • Add MORANS_I_H3 function.
  • Add MORANS_I_QUADKEY function.
  • Add VERSION function.

September 22, 2021

Module h3
Improvement
  • Review HEXRING, KRING functions.
  • Change KRING_INDEXED to KRING_DISTANCES.

September 17, 2021

Module tiler
Fix
  • Fix support for DATE in CREATE_TILESET.
  • Fix naming of internal variables to avoid name collisions.
New
  • Add fraction_dropped_per_zoom in tileset metadata.

September 16, 2021