Sitecore Geo-Spatial Results with Azure Search

This is the second blog post in this series. In the first blog post, I spoke about setting up Azure Search.

In this post, I am going to talk about using a Geo-Spatial lookup query that Azure Search has built in and the associated Index data type called Edm.GeographyPoint

The Azure Search overview can be found here and the configuration document here.

If you’re simply looking for a working example jump over to our GitHub repository that contains the full working helix site.

https://github.com/Aceik/Sitecore-Azure-Search-Spatial

You can also find the latest version of the Sitecore package here:

https://github.com/Aceik/Sitecore-Azure-Search-Spatial/tree/master/lib/Package

Notice on the configuration document that it has a list of EDM data types and it also references the document from Microsoft which has the same list.

A notable exception is the that EDM.GeographyPoint is mentioned in the Microsoft document but not the Sitecore EDM list. So if you need to implement a geo-spatial lookup and get back results based on a latitude and longitude that isn’t possible with the Sitecore Azure Search provider out of the box.

If you dig a little deeper you can see that EDM.GeographyPoint is not in the search code. Have a look at the source code in Sitecore.ContentSearch.Azure.Schema.dll CloudSearchTypeMapper.GetNativeType()

We asked Sitecore support when and if EDM.GeographyPoint would be supported by the Sitecore provider and the answer was no plans in the immediate future.

The solution we came up with to solve this problem essentially involved two steps:

Getting data of type EDM.GeographyPoint into the Azure Search Index.
Performing a Spatial Query on the Azure Search Index (using geo.distance) in a timely manner.

To support both of the above operations we had to do a little digging into the core Sitecore Azure search code. Read on to find out about the solution we came up with or jump on over to the GitHub repository to look over the code.

Getting data into the index

As noted above the EDM.GeographyPoint data type simply isn’t coded into the Sitecore DLLs. You can see this by looking at CloudSearchTypeMapper.GetNativeType() inside the DLL Sitecore.ContentSearch.Azure.Schema.dll.

If you’re still not convinced have a look in the Sitecore.ContentSearch.Azure.Http.dll

The next thing we tried was adding Edm.GeographyPoint as a field converter.

Once again we hit a brick wall with this solution as that doesn’t change the fact that Edm.GeographyPoint is not actually a known cloud type in the core DLLs.

If you look at the error message you get when attempting to add Edm.GeographyPoint as a Converter it tells us that CloudIndexFieldStorageValueFormatter.cs is attempting to look up the native EDM type.

Exception: System.NotSupportedException
Message: Not supported type: ‘Edm.GeographyPoint’
Source: Sitecore.ContentSearch.Azure at Sitecore.ContentSearch.Azure.Schema.CloudSearchTypeMapper.GetNativeType(String edmTypeName) at Sitecore.ContentSearch.Azure.Converters.CloudIndexFieldStorageValueFormatter.FormatValueForIndexStorage(Object value, String fieldName) at Sitecore.ContentSearch.Azure.CloudSearchDocumentBuilder.AddField(String fieldName, Object fieldValue, Boolean append)
….

Working around this limitation took a bit of trial and error but eventually, we nailed it down to the following steps:

Created a new search configuration called spatialAzureIndexConfiguration.
Create a new index configuration based on the new configuration in step 1.

In the search configuration from step 1.

Add the following computed field.

<fields hint="raw:AddComputedIndexField">
 <field fieldName="geo_location" type="Aceik.Foundation.CloudSpatialSearch.IndexWrite.Computed.GeoLocationField, Aceik.Foundation.CloudSpatialSearch" />
 </fields>

Add a new cloud type mapper to map to EDM.GeographyPoint

<cloudTypeMapper ref="contentSearch/indexConfigurations/defaultCloudIndexConfiguration/cloudTypeMapper">
 <maps hint="raw:AddMap">
 <map type="Aceik.Foundation.CloudSpatialSearch.Models.GeoJson, Aceik.Foundation.CloudSpatialSearch" cloudType="Edm.GeographyPoint"/>
 </maps>
 </cloudTypeMapper>

Replace the standard index field storage value formatter with the following the following class.

<indexFieldStorageValueFormatter type="Aceik.Foundation.CloudSpatialSearch.IndexWrite.Formatter.GeoCloudIndexFieldStorageValueFormatter, Aceik.Foundation.CloudSpatialSearch">
 <converters hint="raw:AddConverter">
   ...
 </converters>
 </indexFieldStorageValueFormatter>

Our new formatter GeoCloudIndexFieldStorageValueFormatter.cs inherits from the formatter in the core dlls and overrides the method FormatValueForIndexStorage. It detects if the field name contains “_geo” and basically prevents CloudSearchTypeMapper.GetNativeType from ever running and falling over.

The computed field GeoLocationField returns a GeoJson POCO which is serialised to the GeoJson format (http://geojson.org/) which is required for storage in Azure Search Index.

Getting data out of the index

The GitHub solution demonstrates two solutions.

A direct Azure Search query solution using the query syntax:

"geo.distance(geo_location, geography'POINT(long lat)') le radius"

A LINQ provider that extends the core SearchContext.
- Similar solutions have been outlined for Lucene and solr in the following repositories.
- Sitecore Lucene Spatial
- Sitecore SOLR Spatial

Both solutions above at the end of the day allow you to perform an OData Expression query against your Azure Search Indexes. Using the “geo.distance” filter allows us to perform spatial queries using the computing power of the cloud.

The only downside is that search results returned don’t actually include the distance as a value. They are correctly filtered and ordered by the distance yet the actual distance value itself is not returned. We suggest voting for this change on this ticket :). For now, we suggest using the System.Device.Location.GeoCoordinate to figure out the distance between two coordinates.

References:

In researching for possible solutions already out there we had a look at the following implementations for Lucene and SOLR:

Sitecore Solr Spatial: https://github.com/ehabelgindy/sitecore-solr-spatial
Sitecore Lucene Spatial: https://github.com/aokour/Sitecore.ContentSearch.Spatial

Special Thanks:

Ed Khoze – For the research, he did and help with the initial prototype. Plus google address lookup advice.
Jose Dominguez – For getting the OData filtering query working.

Advanced System Reporter Custom Reports Part 3

Posted by Aceik in Modules, Uncategorized

Simple Custom Parameters Report

For this example, we will build a report which displays a set of results based on a Sitecore Query and custom parameters.

Requirements:

The report will display all items under a path selected by the user in the content tree and only return items of a template selected by the user.

Let’s build it

Step 1: Create a new filter called “Items of Type” and configure as per the image.

Step 2: Create a Parameter item called “TemplateID” and configure as per the image.

Note that the root id refers to the root templates folder you wish the user to be able to filter from.

Step 3: Create a report item called “Items by Template” and configure as per the image.

Step 4: Run the report from ASR. Here you can see we have run the report to return all articles within the categories node of the content tree.

Advanced System Reporter Custom Reports Part 2

Posted by Aceik in Modules, Uncategorized

Simple Sitecore Query based Report

For this example, we will build a report which displays a set of results based on a Sitecore Query.

Requirements:

The report will display all items under a fixed path in the content tree and only return items of predefined specific templates. The report query is fixed and doesn’t require any user input via report parameters.

Let’s build it

Step 1: Create a scanner called “Simple Sitecore Query” and add a query as per the image. The query below gets all items under the categories node where the template is one of the three listed types of templates.

Step 2: Create a new report called “Simple Sitecore Query Report” and configure as per the image.

Step 3: Run the report from ASR. Here you can see we have run the report to return all items of the specific templates within the categories node of the content tree.

Advanced System Reporter Custom Reports Part 1

Posted by Aceik in Modules

Getting started

Firstly download the install package and the source code. https://marketplace.sitecore.net/en/Modules/A/Advanced_System_Reporter.aspx

ASR reports are easy to configure and most of the time require no custom code to be written.

When a custom report requires coding the easiest way is to follow the examples found in the source code. There are also some filters; scanners etc. which are not in use by the default reports that are very handy and you would not know about them unless you have a browse through the ASR projects.

Basics

Report: The report item pulls together the configuration elements required for the ASR report: Scanners, Filters, Viewers, and Commands.
Viewer: The viewer item defines the report view, the columns to display etc.
Scanner: The scanner item defines the search query to populate the viewer.
Parameters: The parameter item defines a parameter input control to be used within a report. It defines what type of control and default values to display or select from depending on control type.
Parameter Type: The parameter type item defines a control type to be used by a parameter for a report. The default Parameter types, for example, are Date picker, Dropdown, Text etc.
Filters: The filter item adds filters to the scanner to provide filtered results for the report. Filters have an attribute field which maps a parameter item to a property within your filter class.
Command: The command item adds commands to the report ribbon. The commands can be used to execute functionality on items in the report.

All Items Scanner

The All items scanner is a great utility scanner. It references the QueryScanner which takes a Sitecore query from the attributes field. This allows you to create a report to output the results of any Sitecore query.