Release notes

October 28th, 2024 (v1.17.0)

Module data

Fix

Module lds

Improvement

  • Added async batch routing services support for TomTom, providing a 2x-3x preformance improvement in processing speed.

Module cpg

Fix

  • Modify RUN_CUSTOMER_SEGMENTATION procedures to overwrite internal temp tables and avoid crashing when called twice in the same session.

Module retail

Fix

Module statistics

Fix

September 24th, 2024 (v1.16.0)

Module tiler

New

  • Add tile_resolution to all tileset creation options. This new setting allows controlling the size of the tiles when rendered in a map. Higher values produce larger tiles.

Module statistics

New

  • Add DETECT_SPATIAL_ANOMALIES and DETECT_SPACETIME_ANOMALIES procedures for detecting anomalous space-time regions in the data. This new capability allows for uncovering unusual patterns in historical spatial or space-time data and for accurately identifying emerging anomalous patterns in space and time.

Fix

Module telco

Fix

August 23rd, 2024 (v1.15.1)

Module statistics

Fix

July 2nd, 2024 (v1.15.0)

Module lds

Fix

  • Fixed a bug where there was more than one matching row in the target for each row in the source. GEOCODE_REVERSE_TABLE now use ST_DISTINCT to solve this issue

Module raster

New

  • Adjustments to use only full-resolution data when the raster table has overviews (blocks of resolution smaller than the block_resolution stated in the metadata)

  • We can now specify per-band no-data values for raster datasets. Rasters bands also now have a ‘colorInterp’ metadata attribute with possible values of red, green, blue, alpha or unknown.

Improvement

  • Optimized raster procedures (RASTER_VALUES, RASTER_AGG_VALUES) to manage bigger data sources. Additionally, the option intersect_center is supported in both procedures to use the centroid of the pixel instead of the contour, in order to speed up the extraction/aggregation for relatively big polygons.

  • New options in RASTER_AGG_VALUES: groupby_vector_columns and groupby_raster_columns to have finer control over the aggregation.

Module import

Fix

  • Import functions in AT Gateway now raise an exception when there is an error. This replaces the previous status message.

Module statistics

New

  • New TIME_SERIES_CLUSTERING procedure, which expects a table containing time series (indexed by a time step column and a unique series identifier) and can group the time series in as many clusters as the user requests.

  • New SPACETIME_HOTSPOTS_CLASSIFICATION procedure. This procedure is designed to analyze spatio-temporal data in order to identify and categorize locations based on their hotspot or coldspot status over time. Utilizing z-score values generated by the Space-Time Getis-Ord function.

Module telco

New

  • New TELCO_PATH_PROFILE procedure, that finds the path between the transmitter (Tx) and receiver (Rx), the obstacles that exist along the propagation path, i.e. those that intersect with the propagation line, and then it evaluates the impact of each obstacle's height

  • New TELCO_PATH_PROFILE_RASTER procedure, an optimised version of the nominal TELCO_PATH_PROFILE procedure which only accepts as inputs a clutter and a terrain elevation rasters loaded using CARTO raster loader or Import API.

  • New CLOSE_IN: This procedure implements a statistical propagation model that describes large-scale propagation path loss over distance at all relevant frequencies in a certain outdor scenario.

  • New EXTENDED_HATA: This new procedure implements the empirical propagation model Extended HATA for radio propagation path loss for the range of frequencies 30 MHz - 3000 MHz in an outdoor scenario.

Module quadbin

Improvement

May 22nd, 2024 (v1.14.0)

Module http_request

New

  • We have a new module named “http_request”. It allows making requests to external endpoints through the function HTTP_REQUEST

Module import

New

  • We have a new module named “import”. It allows importing tables for a given URL containing a file in any of the formats supported by the CARTO Import API. The procedure in charge of this import is IMPORT_URL.

April 18th, 2024 (v1.13.0)

Module raster

New

  • Added new RASTER_VALUES and RASTER_AGG_VALUES procedures:

    • RASTER_VALUES obtains pixel values from a raster where it overlaps with an optional vector data input. It accepts SQL expressions, including operations between bands.

    • RASTER_AGG_VALUES obtains aggregated values from a raster's pixel values where it intersects with a vector data input. It accepts any SQL aggregation expression.

Module clustering

Fix

  • Update ST_CLUSTERKMEANS in order to avoid generating fewer clusters than requested. The maximum number of clusters now will be equal to the number of different points passed.

Module lds

Fix

  • GEOCODE_TABLE not working properly for “geom_column” parameter's value different from “geom”.

March 21st, 2024 (v1.12.0)

Module lds

New

  • All external functions (including those used by the procedures in the lds module) now use a single, unified endpoint. This also simplifies the manual installation of the Analytics Toolbox.

  • GEOCODE_TABLE now uses async batch geocoding with TomTom, boosting the geocoding speed considerably. Also matching by rows is applied ensuring that the address that reaches the TomTom endpoints is as similar as possible as the table value.

Module data

New

  • Add ENRICH_POLYGONS_WEIGHTED procedure. This new procedure enables users to enrich polygons with a provided data query by taking into account the ratio of a specified feature between the intersected and original areas in a way to weigh the contribution of each feature in the enrichment. Note that the original methods only use the geographic area of intersection for this weight.

Module statistics

Fix

Module random

Fix

February 16th, 2024 (v1.11.1)

Module retail

Fix

Module statistics

Fix

January 18th, 2024 (v1.11.0)

Module quadbin

New

  • New QUADBIN_DISTANCE function that returns the distance (number of cells) between two quadbin cells.

Fix

  • Improved precision when calculating cell indexes from latitudes close to the poles (±90º).

Module h3

Fix

  • Fixed an issue with the H3_POLYFILL_TABLE procedure that was not working due to a broken function reference.

Module transformations

New

  • New ST_POINTONSURFACE function that returns a point contained within a given polygon.

Module tiler

Fix

  • Support adding properties from BIGNUMERIC columns in the CREATE_TILESET procedure.

Module data

Fix

  • Fixed an issue when using AVG aggregation, that was not correctly computed when “data_query” contained NULL values.

  • Fixed a problem with DATAOBS_ENRICH_POINTS, that was not parsing vars metadata correctly and producing a crash.

  • Fixed a problem when using “Default” aggregation in DATAOBS_ENRICH_* procedures.

Module lds

Fix

  • Fixed a problem that caused GEOCODE_REVERSE_TABLE crash when a table contains multiple points very close to each other.

Module raster

Improvement

Module statistics

Fix

  • Added a normalization constant for MORANS_I_* procedures

December 21, 2023 (v1.10.0)

Module raster

New

  • New RASTER_VALUE, and RASTER_VALUE_TABLE procedures: updated raster format, introduced performance optimizations and support for custom operations, including multi-band and aggregation expressions.

  • Raster loader v0.4.0: supports new raster format, upload multiple bands and performance optimizations.

Deprecated

  • Previous versions of RASTER procedures in beta

Module retail

Improvement

Module cpg

Improvement

Module tiler

Fix

Module data

Improvement

Deprecated

  • Old quadint/quadkey support in all procedures

November 13, 2023 (v1.9.0)

Module lds

New

  • Add SETUP procedure. This procedure helps configuring the Analytics Toolbox in custom installations. Check this documentation about installing the Analytics Toolbox in your own BigQuery project for more information.

  • Add LDS functions: GEOCODE, GEOCODE_REVERSE, ISOLINE. This functions complement the already existing procedures. They allow geocoding a single address or getting a single isoline without creating a result table.

  • Replace LDS_QUOTA_INFO procedure with GET_LDS_QUOTA_INFO function. This new function allows checking your currently consumed quota with a single query, without the need to store the result as a table.

Fix

Module data

Change

  • Change location of spatial catalog tables: This is only an internal change. Now the catalog is hosted along with the Data Observatory subscriptions. This change decouples the Analytics Toolbox location from the Data Observatory location, providing more flexibility for future developments.

Module statistics

Fix

Module retail

Fix

  • Fix bugs in kring decay procedures: KRING_DECAY, KRING_DECAY_GRID. These procedure are used internally by other procedures. They work finding cells that are within the K-Ring of each cell in a grid, and assigns a weight based on the distance. This helps model changes in a variable across neighboring cells.\

Module cpg

Change

October 13, 2023 (v1.8.0)

Module statistics

New

Module retail

Improvement

Module lds

Improvement

  • New batch service for CREATE_ISOLINES. Used automatically by the procedure. No changes required.

September 8, 2023 (v1.7.0)

Module statistics

New

Module retail

Fix

New

Module lds

Fix

  • Fix GEOCODE_TABLE not iterating through input table rows.

August 7, 2023 (v1.6.0)

Module cpg

New

  • We have added the new procedures CREATE_SPATIAL_SCORE and CREATE_SPATIAL_PERFORMANCE_SCORE to allow users create scores to analyze their network of merchants/point-of-sale based on a business performance metrics or geospatial attributes to analyze their suitability for the CPG brands.

Module statistics

Fix

Module h3

New

  • We have added a new function H3_POLYFILL_MODE that enables different polyfill modes: center, intersects, contains.

Improvement

  • We have improved the performance of H3_POLYFILL by an average factor of 60x.

Module quadbin

New

  • We have added a new function QUADBIN_POLYFILL_MODE that enables different polyfill modes: center, intersects, contains.

Improvement

  • We have improved the performance of QUADBIN_POLYFILL by an average factor of 10x.

July 14, 2023 (v1.5.0)

Module lds

New

  • We are now allowing users to send any of the parameter options supported by the isolines service provider in CREATE_ISOLINES.

  • We have improved our logging of results in GEOCODE_TABLE so more information about processed and geocoded rows is provided to the user. Also the new option “carto_force_geocode” allows the re-geocoding of already geocoded rows.

Fix

  • We have applied some fixes to the CREATE_ROUTES function so the metadata column can be successfully created and informed.

Module data

New

  • We have added the new procedures DATAOBS_ENRICH_POLYGONS_WEIGHTED and DATAOBS_ENRICH_GRID_WEIGHTED. They allow users to enrich polygons or grids based on spatial indexes (e.g. H3, Quadbin) with features from Data Observatory subscriptions by taking into account the ratio of a specified feature between the intersected and original areas in a way to weight the contribution of each feature in the enrichment. Note that the original methods only use the geographic area of intersection for this weight.

Fix

  • We have updated the GRIDIFY_ENRICH procedure so the default value for the “decay” function is taken into consideration.

Module retail

New

Fix

Module statistics

New

Fix

Module quadbin

Change

June 6, 2023 (v1.4.0)

Module lds

New

  • We have created a new procedure named CREATE_ROUTES, which calculates the routes (returned as linestrings) between a set of given origins and destinations (points) in a query. It supports different transport modes and advanced options as the other functions of the lds module.

Module statistics

New

Fix

Module cpg

Fix

  • We have introduced a better way to handle large tables and drop intermediate tables in the UNIVERSE_MATCHING function.

Module tiler

New

May 5, 2023 (v1.3.0)

Module statistics

New

  • Set of procedures to compute Area of Applicability of a model built with BigQuery ML:

    • AREA_OF_APPLICABILITY: This procedure computes the Area of Applicability (AOA) of a Bigquery ML model. It generates a metric which tells the user where the results from a Machine Learning (ML) model can be trusted when the predictions are extrapolated outside the training space (i.e. where the estimated cross-validation performance holds).

    • ENV_BLOCKING: This procedure derives cross validation (CV) folds based on environmental blocking.

Fix

  • We have added a new mandatory parameter called correlation_var when scoring_method = FIRST_PC in the CREATE_SPATIAL_COMPOSITE_UNSUPERVISED

  • We have improved the management of temporary tables in all procedures to compute spatial composites (scores)

Module raster (beta)

Improvement

April 4, 2023 (v1.2.0)

Module statistics

New

  • We have released a new set of procedures within the statistics module to enable users to create spatial scores (also known as composite indicators or indexes) derived from a combination of different features. We have included 3 different procedures:

Module lds

Change

  • We have updated the LDS module to be aligned with the new usage quota model, working as an annual budget and combining consumption across all supported LDS services.

Fix

  • GEOCODE_TABLE now supports null values in the input address column, ignoring them when running the geocoding process to provide results only for the informed addresses.

March 7, 2023 (v1.1.0)

Module raster (beta)

Improvement

Note

The raster module of the Analytics Toolbox for BigQuery v1.1.0 will work only with raster-loader v0.3.0, which pre-computes and stores metadata in the raster table and homogenizes the generic/quadbin raster table format. So, if you update your Analytics Toolbox version to the most recent v1.1.0, please ensure you are using the raster-loader v0.3.0.

  • We have made the RASTER_GETMETADATA faster by retrieving metadata from the table and avoiding creating again the original table.

  • We have improved the RASTER_ST_GETVALUE by removing pixel_geog column and cluster results by quadbin in the quadbin raster format.

  • We have enhanced the RASTER_ST_GETVALUE_FROM_TABLE outcome by preserving the input columns, renaming centroid_quadbin_block and centroid_quadbin_pixel to block_quadbin and quadbin, clustering the results by quadbin in quadbin raster and removing few non-informative columns (e.g. geog and myoffset).

Module cpg

New

  • We have released a new set of functions to enable users to perform the merchant universe matching analysis. It includes 2 different functions, UNIVERSE_MATCHING which performs a fuzzy match between two different POI datasets based on location and name similarity (i.e. one dataset should include current merchants and the other one the complete universe of all potential merchants), and UNIVERSE_MATCHING_REPORT that generates report-like tables summarizing market penetration based on the results of the UNIVERSE_MATCHING analysis.

Fix

Module lds

Improvement

  • From this version, we support line break (\n) and other escaped sequences in addresses in GEOCODE_TABLE.

January 31, 2023 (v1.0.0)

Raster module (beta)

New

Note that Analytics Toolbox for BigQuery v1.0.0 will work only with raster-loader v0.2.0, to use the latest version of rater-loader please ensure you also have access to the latest version of the Analytics Toolbox

Routing module (beta)

Improvement

  • New enhanced version of the routing module with procedures ROUTING_MATRIX to calculate origin-destination matrices and ROUTING_ISOLINES to compute isolines around a set of locations, both supporting multiple transportation modes (car, bike, and walk). These new functions run on top of CARTO’s road network (derived from OSM segments) that is available as a public subscription in the Data Observatory. Please note that these improvements imply breaking changes with the previous version of the routing module.

Module retail

Fix

Module statistics

New

  • Add IDW function (Inverse Distance Weighted).

Module tiler

Improvement

  • Update default zoom values: geo, quadbin, h3 (12, 12, 6).

  • Fix spatial index tiler with empty properties.

December 22, 2022

Module lds

Improvement

  • We have added the possibility to configure more options as parameters when executing the functions to CREATE_ISOLINES. These new options, which depend on the LDS service provider, allow the user to configure more transportation modes such as truck or bike, the possibility of specifying departure or arrival times allowing the creation of reverse isolines, and other options like different routing modes. Please note that these improvements imply breaking changes with the previous version of the CREATE_ISOLINES functions.

  • Added new confidence/relevance metadata to the results of the geocoding function GEOCODE_TABLE. Please note that this improvement has implied breaking changes with the previous versions of the GEOCODE_TABLE and GEOCODE_REVERSE_TABLE functions.

Module cpg

New

  • New FIND_SIMILAR_LOCATIONS function to the CPG module, that allows users to identify which locations (e.g. merchants, stores) are more similar to a chosen location (e.g. top performant) based on the characteristics of their surrounding areas (or trade areas).

Module retail

Improvement

  • Update in BUILD_CANNIBALIZATION_DATA and CANNIBALIZATION_OVERLAP to support buffer, kring and isoline methods. Please note that these improvements have implied some breaking changes in the aforementioned functions invalidating their previous version.

Module data

Fix

  • We have implemented a fix in the weighted averages computation in the GRIDIFY_ENRICH procedure.

Module quadbin

New

  • Quadbin/quadkey conversion functions have been added to ease the conversion from quadbin to quadkey and from quadkey to quadbin indexes.

November 15, 2022

Module retail

Fix

  • __KRING_DECAY_GRID was aggregating the value of the moving window pivoting index instead of neighbour indexes.

November 8, 2022

Module h3

Improvement

  • Add linestrings and points support to function H3_POLYFILL.

November 7, 2022

Module retail

New

  • Add FIND_TWIN_AREAS_WEIGHTED procedure.

November 1, 2022

All modules

New

  • Create release workflows

October 28, 2022

Module s2

New

October 26, 2022

Module transformations

Fix

  • Fix ST_BUFFER crashing with geographies close to the poles.

Module tiler

Improvement

  • Partition tables in spatial index tilesets.

Module cpg

New

  • Add CUSTOMER_SEGMENTATION_ANALYSIS_DATA procedure.

  • Add GENERATE_TRADE_AREAS procedure.

  • Add RUN_CUSTOMER_SEGMENTATION procedure.

Module lds

Fix

  • CREATE_ISOLINES output is stored in the column geom instead of __iso_geom.

Improvement

  • CREATE_ISOLINES output is clustered by geometry.

October 24, 2022

Module processing

Fix

  • Prevent error in ST_VORONOIPOLYGONS, ST_VORONOILINES, ST_VORONOIPOLYGONS, ST_DELAUNAYLINES when points where too close together by rounding input coordinates to 5 decimal places.

October 7, 2022

Module clustering

New

  • Move ST_CLUSTERKMEANS function to core.

Module random

New

  • Move ST_GENERATEPOINTS function to core.

September 30, 2022

Module lds

Improvement

  • Rename LDS_API_URL to API_BASE_URL.

September 29, 2022

Module retail

Improvement

  • LINEAR_REG model is now supported.

September 27, 2022

Module tiler

Improvement

  • Fix bug in __BUILD_REVENUE_MODEL_DATA_GRID: Output table now contains competitor_count.

September 19, 2022

Module tiler

Improvement

  • Set version field in tilers metadata compliant with AT version.

September 17, 2022

Module tiler

Improvement

  • Hotfix: set generate_feature_id to false by default due a bug in the generation.

September 16, 2022

Module data

Improvement

  • Optimize DATAOBS_ENRICH_GRID, DATAOBS_ENRICH_GRID_RAW when the enrich data is a grid compatible with the input.

  • Remove support for s2, geohas and quadbinh in all enrich procedures.

  • Optimize all enrichment result tables by default by clusterizing by the geography or spatial index column.

September 15, 2022

Module s2

New

  • Add S2_CENTER function.

Module retail

Improvement

  • Return shap_values in PREDICT_REVENUE_AVERAGE.

August 26, 2022

Module tiler

New

  • Add generate_feature_id parameter in CREATE_SIMPLE_TILESET.

August 19, 2022

Module retail

Improvement

  • Index parameter admits any type in PREDICT_REVENUE_AVERAGE.

  • Index from revenue_model_data is not casted in PREDICT_REVENUE_AVERAGE to reduce query data processing.

August 16, 2022

Module tiler

Fix

  • Fixed bug when parsing zoom columns in geography tilers.

August 11, 2022

Module tiler

Improvement

  • Unify extra_metadata into metadata in tiler metadata.

August 9, 2022

Module h3

Fix

  • Apply make_valid in H3_BOUNDARY.

August 8, 2022

Module tiler

Fix

  • Fixed bug in metadata CREATE_SPATIAL_INDEX_TILESET.

August 5, 2022

Module retail

Improvement

  • Check fixed model options in BUILD_REVENUE_MODEL procedure.

August 3, 2022

Module retail

New

  • Add BUILD_CANNIBALIZATION_DATA procedure.

  • Add CANNIBALIZATION_OVERLAP procedure.

July 26, 2022

Module lds

New

  • Add GEOCODE_TABLE procedure.

  • Add LDS_QUOTA_INFO procedure.

  • Add GEOCODE_REVERSE_TABLE procedure.

  • Add CREATE_ISOLINES procedure.

July 21, 2022

Module statistics

Improvement

  • Added HH, LL, LH, HL detection in LOCAL_MORANS_I functions.

July 20, 2022

Module tiler

Fix

  • Fixed bug when passing multiline input to CREATE_SPATIAL_INDEX_TILESET.

July 8, 2022

Module data

New

  • Add quadbin support to module.

Module retail

New

  • Add quadbin support

Module tiler

New

  • Support of quadgrid as alias in CREATE_POINT_AGGREGATION_TILESET.

Improvement

  • Use core functions to convert H3 from int to string.

July 7, 2022

Module h3

Fix

  • Correctly handle large polygons in H3_POLYFILL.

  • Fixed wrong uppercase for quadbin and h3 tile ids

Module statistics

New

  • Add SMOOTHING_MRF_QUADBIN procedure.

  • Add LOCAL_MORANS_I_QUADBIN function.

  • Add MORANS_I_QUADBIN function.

  • Add GETIS_ORD_QUADBIN function.

  • Quadbin support in GWR_GRID function.

Module tiler

Fix

  • Fixed tablename quoting in BigQuery tiler

July 6, 2022

Module retail

Fix

  • Fix quadkey support in FIND_WHITESPACE_AREAS

June 29, 2022

Module retail

Fix

  • Fixing bugs when not providing store variables to BUILD_REVENUE_MODEL_DATA

June 23, 2022

Module quadbin

New

  • Add QUADBIN_BBOX function.

  • Add QUADBIN_BOUNDARY function.

  • Add QUADBIN_CENTER function.

  • Add QUADBIN_FROMGEOGPOINT function.

  • Add QUADBIN_FROMLONGLAT function.

  • Add QUADBIN_FROMZXY function.

  • Add QUADBIN_ISVALID function.

  • Add QUADBIN_KRING function.

  • Add QUADBIN_KRING_DISTANCES function.

  • Add QUADBIN_POLYFILL function.

  • Add QUADBIN_RESOLUTION function.

  • Add QUADBIN_SIBLING function.

  • Add QUADBIN_TOCHILDREN function.

  • Add QUADBIN_TOPARENT function.

  • Add QUADBIN_TOZXY function.

June 21, 2022

Module data

Fix

  • Fix BQ inplace enrichment with uppercase column names.

Module tiler

Improvement

  • Support H3 integer in CREATE_SPATIAL_INDEX_TILESET.

June 13, 2022

Module data

Fix

  • Fix order of __KRING_DECAY parameters.

June 10, 2022

Module tiler

Fix

  • Fix global stats in CREATE_SPATIAL_INDEX_TILESET.

Improvement

  • Check input resolution limits in CREATE_SPATIAL_INDEX_TILESET.

June 9, 2022

Module data

Fix

  • Report missing subscription geographies in DATAOBS_ENRICH_ procedures.

June 6, 2022

Module data

Fix

  • Fix stability issues in GRIDIFY_ENRICH procedure.

June 3, 2022

Module retail

New

  • Add store features and competitors stores features into BUILD_REVENUE_MODEL_DATA procedure.

Fix

  • Fixing bugs calling other routines

June 2, 2022

Module tiler

New

  • Add spatial index tileset generator CREATE_SPATIAL_INDEX_TILESET.

May 16, 2022

Module tiler

Fix

  • Bug fixing when the geometry early collapse

April 27, 2022

Module data

New

  • Add GRIDIFY_ENRICH procedure.

Module retail

New

  • Add FIND_TWIN_AREAS procedure.

April 22, 2022

Module data

New

  • Allow using tables as input, not just queries.

  • Add in place enrichment.

  • Allow using tables as input.

  • Accept NULL as output.

April 20, 2022

Module retail

Fix

  • Fix BUILD_REVENUE_MODEL_DATA k-ring buffer computation.

March 21, 2022

Module transformations

Improvement

  • ST_CONCAVEHULL now allows arrays with one/two points as input.

Module retail

New

  • Add COMMERCIAL_HOTSPOTS procedure.

Module statistics

New

  • Add SMOOTHING_MRF_H3 procedure.

  • Add SMOOTHING_MRF_QUADKEY procedure.

February 16, 2022

Module statistics

New

  • Add VARIOGRAM function.

  • Add ORDINARY_KRIGING function.

  • Add ORDINARY_KRIGING_TABLE procedure.

February 15, 2022

Module h3

New

  • Add H3_CENTER function.

  • Add H3_RESOLUTION function.

Module retail

New

  • Add FIND_WHITESPACE_AREAS procedure.

February 3, 2022

Module retail

New

  • Remove features importance tables from BUILD_REVENUE_MODEL procedure.

February 1, 2022

Module statistics

New

  • Add LOCAL_MORANS_I_H3 function.

  • Add LOCAL_MORANS_I_QUADKEY function.

Improvement

  • Improve MORANS_I_H3 and MORANS_I_QUADKEY implementation.

January 10, 2022

Module statistics

Fix

  • Replace STDDEV by STDDEV_POP in GETIS_ORD_H3 and GETIS_ORD_QUADKEY.

December 22, 2021

Module retail

New

  • Add BUILD_REVENUE_MODEL_DATA procedure.

  • Add BUILD_REVENUE_MODEL procedure.

  • Add PREDICT_REVENUE_AVERAGE procedure.

December 16, 2021

Module accessors

Improvement

  • Deployment schema "carto" instead of "accessors".

Removed

  • Remove VERSION function.

Module constructors

Improvement

  • Deployment schema "carto" instead of "constructors".

Removed

  • Remove VERSION function.

Module geohash

Improvement

  • Deployment schema "carto" instead of "geohash".

Removed

  • Remove VERSION function.

Module h3

Improvement

  • Deployment schema "carto" instead of "h3".

  • Rename ST_ASH3 function to H3_FROMGEOGPOINT.

  • Rename LONGLAT_ASH3 function to H3_FROMLONGLAT.

  • Rename ST_ASH3_POLYFILL function to H3_POLYFILL.

  • Rename ST_BOUNDARY function to H3_BOUNDARY.

  • Rename ISVALID function to H3_ISVALID.

  • Rename COMPACT function to H3_COMPACT.

  • Rename UNCOMPACT function to H3_UNCOMPACT.

  • Rename TOPARENT function to H3_TOPARENT.

  • Rename TOCHILDREN function to H3_TOCHILDREN.

  • Rename ISPENTAGON function to H3_ISPENTAGON.

  • Rename DISTANCE function to H3_DISTANCE.

  • Rename KRING function to H3_KRING.

  • Rename KRING_DISTANCES function to H3_KRING_DISTANCES.

  • Rename HEXRING function to H3_HEXRING.

Removed

  • Remove VERSION function.

Module measurements

Improvement

  • Deployment schema "carto" instead of "measurements".

Removed

  • Remove ST_ANGLE, already present in Bigquery.

  • Remove VERSION function.

Module placekey

Improvement

  • Deployment schema "carto" instead of "placekey".

  • Rename H3_ASPLACEKEY function to PLACEKEY_FROMH3.

  • Rename PLACEKEY_ASH3 function to PLACEKEY_TOH3.

  • Rename ISVALID function to PLACEKEY_ISVALID.

Removed

  • Remove VERSION function.

Module processing

Improvement

  • Deployment schema "carto" instead of "processing".

Removed

  • Remove VERSION function.

Module s2

Improvement

  • Deployment schema "carto" instead of "s2".

  • Rename ID_FROMHILBERTQUADKEY function to S2_FROMHILBERTQUADKEY.

  • Rename HILBERTQUADKEY_FROMID function to S2_TOHILBERTQUADKEY.

  • Rename LONGLAT_ASID function to S2_FROMLONGLAT.

  • Rename ST_ASID function to S2_FROMGEOGPOINT.

  • Rename ID_FROMTOKEN function to S2_FROMTOKEN.

  • Rename TOKEN_FROMID function to S2_TOTOKEN.

  • Rename ID_FROMUINT64REPR function to S2_FROMUINT64REPR.

  • Rename UINT64REPR_FROMID function to S2_TOUINT64REPR.

  • Rename ST_BOUNDARY function to S2_BOUNDARY.

Removed

  • Remove VERSION function.

Module transformations

Improvement

  • Deployment schema "carto" instead of "transformations".

Removed

  • Remove VERSION function.

Module clustering

Improvement

  • Deployment schema "carto" instead of "clustering".

Removed

  • Remove VERSION function.

Module data

Improvement

  • Deployment schema "carto" instead of "data".

Removed

  • Remove VERSION function.

Module random

Improvement

  • Deployment schema "carto" instead of "random".

Removed

  • Remove VERSION function.

Module routing

Improvement

  • Deployment schema "carto" instead of "routing".

Removed

  • Remove VERSION function.

Module statistics

Improvement

  • Deployment schema "carto" instead of "statistics".

Removed

  • Remove VERSION function.

Module tiler

Improvement

  • Deployment schema "carto" instead of "tiler".

Removed

  • Remove VERSION function.

December 13, 2021

Module tiler

Improvement

  • Add support for legacy project names that contain dots or other special characters.

December 10, 2021

Module statistics

New

  • Add KNN function.

  • Add GFUN function.

  • Add LOF function.

  • Add LOF_TABLE procedure.

November 30, 2021

Module tiler

Fix

  • Escape single-quoted properties in the CREATE_TILESET metadata when obtaining the geometry type.

November 25, 2021

Module statistics

New

  • Add GWR_GRID function.

November 16, 2021

Module tiler

Fix

  • Use zoom_min_column as zoom_min and zoom_max_column as zoom_max when provided.

Improvement

  • Optimize simplification method instead of the native BigQuery.

November 5, 2021

Module data

Improvement

  • Fields named dimension, total, intersection and input_area are now __carto_dimension, __carto_total and __carto_intersection and __carto_input_area. Also the column _carto_enrichment_ is now __carto_enrichment. This affects all the raw enrichment procedures: ENRICH_POINTS_RAW, ENRICH_POLYGONS_RAW, ENRICH_GRID_RAW, DATAOBS_ENRICH_POINTS_RAW, DATAOBS_ENRICH_POLYGONS_RAW, DATAOBS_ENRICH_GRID_RAW.

Fix

  • User provided queries can now have columns named dimension, total, intersection, input_area, _nonglobal, which could have collided previously with internal columns. All internal columns are now prefixed with __carto_. This affects all the enrichment procedures: ENRICH_POINTS, ENRICH_POLYGONS, ENRICH_GRID, DATAOBS_ENRICH_POINTS, DATAOBS_ENRICH_POLYGONS, DATAOBS_ENRICH_GRID, ENRICH_POINTS_RAW, ENRICH_POLYGONS_RAW, ENRICH_GRID_RAW, DATAOBS_ENRICH_POINTS_RAW, DATAOBS_ENRICH_POLYGONS_RAW, DATAOBS_ENRICH_GRID_RAW.

November 2, 2021

Module data

Improvement

  • Add associated_geography_slug column to DATAOBS_SUBSCRIPTIONS.

  • Some column names have changed in DATAOBS_SUBSCRIPTIONS:

    • table is now dataset_table

    • associated_geography is now associated_geography_table

  • Some column names have changed in DATAOBS_SUBSCRIPTION_VARIABLES:

    • geography_slug is now associated_geography_slug.

  • Some column names have changed in DATAOBS_SAMPLES:

    • table is now dataset_sample_table.

  • In DATAOBS_SUBSCRIPTION_VARIABLES, the column associated_geography_slug is now NULL for geography datasets.

Fix

  • The output of DATAOBS_SUBSCRIPTIONS and DATAOBS_SUBSCRIPTION_VARIABLES now omits items that are not Data Observatory products.

October 15, 2021

Module data

Fix

  • Enrichment results could mix input rows and values, specially for large inputs due to multiple evaluation of nondeterministic ROW_NUMBERS expression.

Improvement

  • New API, with consistent naming of procedures `

September 23, 2021

Module s2

Improvement

  • Rename functions ID_FROMUINT64REPR, UINT64REPR_FROMID to follow convention.

Module statistics

New

  • Create statistics module.

  • Add GETIS_ORD_H3 function.

  • Add GETIS_ORD_QUADKEY function.

  • Add MORANS_I_H3 function.

  • Add MORANS_I_QUADKEY function.

  • Add VERSION function.

September 22, 2021

Module h3

Improvement

  • Review HEXRING, KRING functions.

  • Change KRING_INDEXED to KRING_DISTANCES.

September 17, 2021

Module tiler

Fix

  • Fix support for DATE in CREATE_TILESET.

  • Fix naming of internal variables to avoid name collisions.

New

  • Add fraction_dropped_per_zoom in tileset metadata.

September 16, 2021

Module data

Changes

  • Support sub_ (table) subscriptions in:

    • DATAOBS_ENRICH_POINTS

    • DATAOBS_ENRICH_POINTS_WITH_MEASURES

    • DATAOBS_ENRICH_POLYGONS_WITH_AGGREGATION

    • DATAOBS_ENRICH_POLYGONS_WITH_MEASURES

    • DATAOBS_ENRICH_GRID

September 14, 2021

Module s2

Fix

  • Avoid keeping planar shape in spherical coordinates in ST_BOUNDARY.

September 3, 2021

Module data

Changes

  • Search for tables in DATAOBS_SAMPLES, DATAOBS_SUBSCRIPTIONS, DATAOBS_SUBSCRIPTIONS_VARIABLES.

August 31, 2021

Module data

Improvement

  • DATAOBS_ENRICH_* procedures are now optimized to reduce the amount of data processed by taking advantage of clusterization by geography in the Data Observatory.

  • The input_id_column parameter has been removed from the following procedures:

    • DATAOBS_ENRICH_POINTS

    • DATAOBS_ENRICH_POINTS_WITH_MEASURES

    • DATAOBS_ENRICH_POLYGONS_WITH_AGGREGATION

    • DATAOBS_ENRICH_POLYGONS_WITH_MEASURES

    • ENRICH_POINTS

    • ENRICH_POINTS_WITH_MEASURES

    • ENRICH_POLYGONS_WITH_AGGREGATION

    • ENRICH_POLYGONS_WITH_MEASURES

August 26, 2021

Module data

Improvement

  • ENRICH_GRID and DATAOBS_ENRICH_GRID procedures take input_query and input_index_column parameters instead of the indices array.

  • DATAOBS_ENRICH_* procedures now report missing subscriptions in a more user-friendly way.

Module routing

Improvement

  • Pass network to JS UDF natively

  • Support custom speed over LineStrings for GENERATE_NETWORK function.

  • Support custom speed over LineStrings for FIND_SHORTEST_PATH function.

  • Support custom speed over LineStrings for DISTANCE_MAP function.

August 25, 2021

Module tiler

Fix

  • Fix support for TIMESTAMP,DATETIME in CREATE_TILESET and CREATE_SIMPLE_TILESET.

  • Reduce parallelization to skip the recent error: "Not enough resources for query planning - too many subqueries or query is too complex"

August 24, 2021

Module h3

Fix

  • Support GEOMETRYCOLLECTION from ST_ASH3_POLYFILL.

August 6, 2021

Module data

New

  • Add DATAOBS_ENRICH_GRID procedure.

  • Add ENRICH_GRID procedure.

August 4, 2021

Module h3

New

  • Add KRING_INDEXED function.

July 30, 2021

Module geohash

New

  • Create geohash module.

  • Add VERSION function.

  • Add ST_BOUNDARY function.

Module tiler

Improvement

  • Checking if the output table exists before running the tiler.

July 13, 2021

Module data

Improvement

  • DATAOBS_ENRICH_* procedures take single source parameter with default carto-customers project.

July 9, 2021

Module data

New

  • Add DATAOBS_SAMPLES procedure.

Improvement

  • DATAOBS_SUBSCRIPTIONS and DATAOBS_SUBSCRIPTION_VARIABLES take single source parameter with default carto-customers project. Views with sub_ prefix are supported.

Module routing

New

  • Create routing module.

  • Add GENERATE_NETWORK function.

  • Add GENERATE_NETWORK_TABLE procedure.

  • Add FIND_SHORTEST_PATH function.

  • Add FIND_SHORTEST_PATH_FROM_NETWORK function.

  • Add FIND_SHORTEST_PATH_FROM_NETWORK_TABLE procedure.

  • Add DISTANCE_MAP function.

  • Add DISTANCE_MAP_FROM_NETWORK function.

  • Add DISTANCE_MAP_FROM_NETWORK_TABLE procedure.

  • Add VERSION function.

July 8, 2021

Module data

Fix

  • Fix DATAOBS_SUBSCRIPTIONS and DATAOBS_SUBSCRIPTION_VARIABLES not working with subscriptions in which the provider id contained underscores.

July 1, 2021

Module tiler

New

  • Add new automatic tileset generator CREATE_TILESET.

June 23, 2021

Module data

New

  • Add DATAOBS_SUBSCRIPTIONS procedure.

  • Add DATAOBS_SUBSCRIPTION_VARIABLES procedure.

Improvement

  • Change spatial catalog structure using two tables.

  • Improve enrichment procedures:

    • Use public geographies instead of subscription views

    • Remove unnecessary variables from ENRICH_POLYGONS_WITH_MEASURES

    • Make input_id_column always mandatory

    • Add input_area consistently

June 2, 2021

Module data

New

  • Create data module.

  • Add VERSION function.

  • Add DATAOBS_ENRICH_POINTS procedure.

  • Add DATAOBS_ENRICH_POINTS_WITH_MEASURES procedure.

  • Add DATAOBS_ENRICH_POLYGONS_WITH_AGGREGATION procedure.

  • Add DATAOBS_ENRICH_POLYGONS_WITH_MEASURES procedure.

  • Add ENRICH_POINTS procedure.

  • Add ENRICH_POINTS_WITH_MEASURES procedure.

  • Add ENRICH_POLYGONS_WITH_AGGREGATION procedure.

  • Add ENRICH_POLYGONS_WITH_MEASURES procedure.

June 1, 2021

Module s2

New

  • Add TOKEN_FROMID function.

  • Add ID_FROMTOKEN function.

  • Add ID_FROM_UINT64REPR function.

  • Add UINT64REPR_FROM_ID function.

May 19, 2021

Module tiler

Improvement

  • Use semantic versioning.

  • Add features_count and geometry labels to the tilesets.

May 6, 2021

Module clustering

Improvement

  • Change ST_CLUSTERKMEANS returning cluster and geom

May 4, 2021

Module accessors

New

  • Create accessors module.

  • Add ST_ENVELOPE function.

  • Add VERSION function.

Module processing

New

  • Create processing module.

  • Add ST_VORONOIPOLYGONS function.

  • Add ST_VORONOILINES function.

  • Add ST_DELAUNAYPOLYGONS function.

  • Add ST_DELAUNAYLINES function.

  • Add ST_POLYGONIZE function.

  • Add VERSION function.

Module transformations

New

  • Add ST_CONCAVEHULL function.

April 30, 2021

Module clustering

New

  • Create clustering module.

  • Add ST_CLUSTERKMEANS function.

  • Add VERSION function.

April 29, 2021

Module constructors

New

  • Add ST_BEZIERSPLINE function.

  • Add ST_MAKEELLIPSE function.

Module measurements

New

  • Create measurements module.

  • Add ST_ANGLE function.

  • Add ST_AZIMUTH function.

  • Add ST_MINKOWSKIDISTANCE function.

Module transformations

New

  • Rename module to transformations.

  • Add ST_CENTERMEAN function.

  • Add ST_CENTERMEDIAN function.

  • Add ST_CENTEROFMASS function.

  • Add ST_DESTINATION function.

  • Add ST_GREATCIRCLE function.

  • Add ST_LINE_INTERPOLATE_POINT function.

Module tiler

Removed

  • Remove quota & project checking.

April 28, 2021

Module constructors

New

  • Create constructors module.

  • Add ST_MAKEENVELOPE function.

  • Add ST_TILEENVELOPE function.

  • Add VERSION function.

April 16, 2021

Module transformations

New

  • Create transformation module.

  • Add ST_BUFFER function.

  • Add VERSION function.

April 10, 2021

Module random

Improvement

  • ST_GENERATEPOINTS now uses a spherically uniform distribution. Previously used to by uniform on projection.

April 9, 2021

Module h3

Improvement

  • Use hexadecimal as default type instead of int for h3 indexes.

Fix

  • Fix ST_BOUNDARY generating error when not able to parse geometry.

Module placekey

Improvement

  • Placekey conversions works with hexadecimal h3 indexes instead of int.

Module random

New

  • Create random module.

  • Add ST_GENERATEPOINTS function.

  • Add VERSION function.

March 31, 2021

Module h3

New

  • Create h3 module.

  • Add ST_ASH3 function.

  • Add LONGLAT_ASH3 function.

  • Add ST_ASH3_POLYFILL function.

  • Add ST_BOUNDARY function.

  • Add ISVALID function.

  • Add COMPACT function.

  • Add UNCOMPACT function.

  • Add TOPARENT function.

  • Add TOCHILDREN function.

  • Add ISPENTAGON function.

  • Add DISTANCE function.

  • Add KRING function.

  • Add HEXRING function.

  • Add VERSION function.

Module placekey

New

  • Create placekey module.

  • Add H3_ASPLACEKEY function.

  • Add PLACEKEY_ASH3 function.

  • Add ISVALID function.

  • Add VERSION function.

Module s2

New

  • Create s2 module.

  • Add ID_FROMHILBERTQUADKEY function.

  • Add HILBERTQUADKEY_FROMID function.

  • Add LONGLAT_ASID function.

  • Add ST_ASID function.

  • Add ST_BOUNDARY function.

  • Add VERSION function.

Module tiler

New

  • Add QUOTA procedure in order to display the monthly quota and quota available.

  • Implement support for layer name.

  • Add a max limit to the max_tile_size_kb property. Adds a new option (skip_validation) to circunvent this limit, which also allows going over the max partition limit.

  • Add a limit to the max amount of expected tiles at the highest zoom level for Simple Tilesets. It can be disabled via the skip_validation option of modified (for tests using max_tiles_at_maxzoom).

  • Add new drop_fraction_as_needed strategy initial implementation. Also adds internal debug and fraction_drop_per_zoom_level options to help debugging.

  • Add new metadada input parameter to set the properties name, description, legends and also extra_metadata.

  • Add zoom_min_column and zoom_max_column to control the zoom level at which particular features are visible.

Improvement

  • Change the default and max partition limit to 3999.

  • Make aggregation tilesets to use the quadkey module UDF LONGLAT_ASQUADINTLIST_RESOLUTION in order to generate quadkey aggregations.

Removed

  • Removed zoom_step option.

Fix

  • Fix bug that left empty the generator options in the metadata cell.

  • Fix CREATE_SIMPLE_TILESET failing with single column tables.

  • Fix collision with frequency column name.

February 10, 2021

Module tiler

New

  • Add tileset type to metadata and BigQuery labels.

  • Add the prefix "aggregation_" to aggregation options: type, resolution and placement.

Improvement

  • Moved to the Advanced Spatial Extension.

  • Change "max_tile_size_strategy" default to "throw_error", matching the other defaults and improve the exception error message in CreatePointAggregationTileset.

  • Rename adapting to BigQuery conventions, using UPPER CASE and separating words with underscore "_".

  • Set up limits for the procedures based on a limits table in the config project. The users can only execute the tiler from the "billing_project" and they have a maximum amount of processed bytes per month.

Removed

  • Remove compression levels 10-12 to reduce library size.

  • Remove ST prefix from functions not using geographies.

  • Remove unused functions like st_tileenvelope or st_tileenvelope_bbox.

Last updated