One Performance Blog to Rule them all – Combining the 6 Pillars of Speed

I have done a number of posts and talks at user groups on Page Speed and performance over the last few years. I have split the various topics into individual blog posts for the most part as performance is dependent on many factors. What has really been missing is a complete demo of how all the different techniques come together to give your site a really good score. So that’s what I intend to demo here is the combination of the 6 pillars of page speed in one Sitecore instance. To recap here are the 6 pillars of page speed performance in my opinion:

1) Introduce image lazy loading

2) Ensure a cache strategy is in place and verify its working. (must have adequately sized production servers)

3) Deploy image compression techniques

4) Use responsive images (must serve up smaller images sizes for mobile)

5) Introduce Critical CSS and deferred CSS files

6) Javascript is not a page speed friend. Defer Defer Defer

I have shown a subset of these previously but crucially three critical pillars to do with imaging were hard to achieve at the time. This is now possible due to being able to support Next Gen image compression (webp), which I wrote about in my previous blog. With a little more time and investigation Image Lazy Loading, responsiveness and image compression to give a more complete picture of how each pillar impacts page speed.

Here are the tools and blogs I will use to achieve each of these:

1) Image Lazy Loading – Blog post by MVP Sitecore SAM and https://github.com/thinker3197/progressively

2) SXA Cache Settings – SXA official documentation

3) Next Image (WEBP) Image Compression – https://github.com/Aceik/ImageCompression

4) SXA Responsive Images – SXA official documentation

5) Introduce Critical CSS and deferred CSS files – https://github.com/Aceik/Sitecore-Speedy

6) Javascript is not a page speed friend. Defer Defer Defer – https://github.com/Aceik/Sitecore-Speedy

Alternatives: Mark Gibbons (MVP) recently upgraded the Dianoga image library to support WEBP. Worth a look if you don’t want to use a third party API. It also supports a CDN. Also Vincent Lui (MVP) also pointed out in his recent SUGCON talk, you can achieve both image compression and image lazy loading via some of the modern CDN’s. That is a great (easy) option if you are retro fitting these techniques to a live website.

I’m not going to dive deep into exactly how to setup each of these things as I think the individual links have sufficient instructions. I will show in the Demo videos how each pillar impacts the HTML rendered. For the most part I am keen to demonstrate the impact of each of these line items and how each one will benefit your page speed score.

Before we begin its important to understand that the algorithm (Lighthouse) behind Google’s Page Speed insight doesn’t work in an exactly linear fashion. If you improve your score by ticking off one of the above, don’t expect ticking off another issue will have the same benefit. The last 20 points out of 100 (on the mobile scoring system) is that hardest to achieve based on what I have seen.

Live Demo Video Series that accompanies this blog:


Test Outline:

Google Page Speed Insights — Scores can fluctuate widely based on network latency. At time you will experience score fluctuations at different times of the day on the same site.

In general this is a guide

Here is the general outline of the VM that hosted the IIS instance for testing. I also put the VM under some basic load while running the tests.

  • All the test below used Sitecore 9.3 and the SXA habitat example site.
  • Test used the live Google Page Speed insights tool via the url: https://developers.google.com/speed/pagespeed/insights/
  • Sitecore was setup on an Azure VM with the specifications:undefined
  • The test was run 5 times, to get an average score.
  • The test page was the homepage of the Habitat site and the page was requested before running the test 5 times so that the instance could be considered warm.
  • EXM and XDB were not running on these test instances.
  • Test results are Mobile Page Speed scores only – This is the most important metric in today environment and good desktop scores are not really a challenge.
  • The default Habitat cache rendering for Navigation was left on for all tests. (without this the site fails under basic load altogether)
  • All tests were conducted under load in an attempt to replicate a production environment. For this I used a node package called loadtest.
  • SXA CSS/Javascript optimisations were turned on, but as I have mentioned before this has a minimal performance boost.

loadtest -c 10 –rps 10 http://baselinecd.dev.local/

10 requests per second with a concurrency of 10

Baseline Score

The Baseline score encompasses the habitat site installed with no modifications.

Result: 48 / 38 / 40 / 34 / 38 = 39.6/100 Average

Observation: Heavily penalised for CSS and Javscript loading times.


Image Lazy Loading

All images on the homepage were converted to be Lazily loaded. A single large blurred image was used as the placeholder for all images.

Result: 57 / 55 / 61 / 52 / 63 = 57.6/100 Average

Observation: Around the mid point of the scale, image lazy loading has around a 15 – 20 point impact.


Rendering Cache Strategy

I have blogged extensively about this in the past but setting up cache settings properly is so critical and has a major impact. Its also one of the easiest things to fix for a poorly performing Sitecore site. Also note the only way to accurately demonstrate the impact that Rendering cache has on a site is to test it under load.

This test was run with higher user per second: loadtest -c 10 –rps 30 http://baselinecd.dev.local/

With Cache Enabled:

49 / 56 / 41 / 54 / 54 = 50.8

Without Cache Enabled:

ERR_TIMED_OUT / ERR_TIMED_OUT / ERR_TIMED_OUT / ERR_TIMED_OUT / ERR_TIMED_OUT = You get the point ūüôā

Observation: Rendering cache settings are critical and should be the first step in Page Load Speed refinement for a Sitecore site. 10 Point benefit observed once a site is stable under load.


Image Compression

Result: 60 / 58 / 61 / 62 / 62 = 60.6/100 Average

Observation: Around the mid point of the scale, image lazy loading has around a 20 point impact.


Critical CSS

Result: 74 / 78 / 79 / 81 / 81 = 78.6/100 Average

Observation: The combination of critical CSS in the head and Deferred styles provides a meaningful page speed boost. 25 Point observed benefit.


Deferred Javascript

Result: 92 / 94 / 93 / 94 / 94 = 93.4/100 Average

Observation: Javascript has a massive impact, reducing it drastically in the initial payload provides massive page speed improvements. 40 Point observed benefit.

You might think, hey I will just do Deferred Javascript and it will be all good. While this particular PIllar/Criteria does have the biggest impact. Every site is different and as mentioned earlier scores fluctuate. The upper part of the scoring system is the hardest to reach. So while this is a great starting point, ignore the other speed pillars at your peril.


Responsive Images

Result: 56 / 54 / 59 / 56 / 60 = 57/100 Average

Observation: Around the mid point of the scale converting images to be responsive (srcset support) has about a 10 point impact.


Results Summary

CriteriaAverage ScoreObserved Benefit
No Change (SXA Habitat Home OOTB)41.8 / 100
Image Lazy Loading57.6 / 10015 Points
Sitecore Rendering/HTML Cache Settings50.8 / 10010 Points
Image Compression (webp)60.6 / 10020 Points
Critical CSS78.6 / 10025 Points
Deferred Javascript93.4 / 10040 Points
Responsive Images57 / 10010 Points

The Pillars Combined

In isolation we can see the rough results of what each of the pillars might do to our Page Speed. The real question is what does combining all these pillars produce.

Result: 100 / 100 / 100 / 100 / 100 = 100/100 Average

Observation: Do I expect this on an actual production site realistically ? That is certainly the dream, but in reality you should be over the moon if you make it into the 90s and pat your self on the back if you get into the 80s as well. For any Sitecore site if you make it into the 90’s for mobile, your doing an amazing job.

Admittedly for the combined demo I skipped the responsive image pillar. SXA supports Responsive Images but not in combination with data attributes. It was going to be a bunch of work to write a custom SXA handler to support both lazy loading and responsiveness at the same time. That is not to say its not possible. Either way the impact was minimal.

Conclusion

Page speed is so critical to SEO and visitor conversion. A slow site instantly turns away users on mobile and tablet devices. Admittedly the final result shown above and in the video have required that all the right tools be available to the Sitecore community. Which up until recently you likely needed to bake your own solutions in order to get that over the line.

I think its now becoming possible to aim fairly high (90/100 on mobile) with our Page Speed scores, but it does require getting most if not all of the Architecture Pillars above working together. Its worth learning each of these and understanding the pitfalls and limitations if you want really great page speed. Good luck and feel free to get in touch with any questions.

Footnote

The combined pillars can produce great results but you still need to load test before going live. Checkout the video below where I search for the breaking point using the loadtest tool. Please note that this node based load test tool should just be used for a guide. Before go live I recommend using a hosted load tool solution that has multiple geographic locations. Tests done based on one network location or device will result in a network bottle neck and give you false positives.

Bonus Video: https://www.youtube.com/watch?v=96YcxyhYh0U

Next Gen Image Compression in Sitecore

Spoiler: This post is not a post about Dianoga, I take a deep dive into Tiny PNG and Kraken.IO integrations into Sitecore. The results are worth checking out at the bottom.


At the start of the year, I’ve picked up where I left off, on page speed. Last year I took a deep dive into attempting to improve the page speed on Sitecore SXA sites by using some of Google’s recommended techniques to structure the page. If you haven’t already seen it, head on over the Sitecore Speedy and see some of the results we achieved.

I’ll be the first to admit that getting really good page speed scores isn’t easy. It takes a lot of different factors to come together. Just as a reminder, here is the main list that I would consider you need to check off to be winning at this game.

1) Introduce image lazy loading

2) Ensure a cache strategy is in place and verify its working.

3) Dianoga is your friend for image compression

4) Use responsive images (must serve up smaller images sizes for mobile)

5) Introduce Critical CSS and deferred CSS files

6) Javascript is not a page speed friend. Defer Defer Defer

For this post, i’m going to look at an alternative to Dianoga. I’m a big fan of Dianoga and have used it over the years to crunch loads of oversized images introduced by Content Editors. I will, however, say that it can add complexity to deployments and CI/CD pipelines and while some claim to have had success in Azure Apps, others have not.

On the flip side, content editors love Tiny PNG, which is one of the most popular image compression website utilities going around. Tiny PNG also has a developer API, so we have used this to build in a compression tool that can be used directly from your Sitecore toolbar.

The button below is hooked up to chat to Tiny PNG API. It will send across your image data and receive a compressed image back for storage.


Full disclosure, I’m not the first person to hook up Tiny PNG to the image library. I could find two other implementations

One will allow you to run a powershell script to connect to the Tiny PNG API and the other is a module to connect to the API on upload.


This implementation of the Tiny PNG API introduces the following variances:

  • A button in the CMS to crunch any single image.
  • A scheduled task that will process any image not already processed.
  • Error handling for when the API limits are reached
  • Logging that outlines which images were processed.
  • Before and After compression information stored in any Image field of choice.
  • A feature toggle to turn the whole feature on/off

All the source code is available at: https://github.com/Aceik/ImageCompression

Now let’s jump in have a look at the results just from crunching a few images down:

Without image compression:

Click to Enlarge Image

To compress the images on the page, we head on over to the “Compress” button in the Media tab that we have introduced.

Click to enlarge

A few examples of compression results taken from homepage images:

Before: 158.4 KB | After: 110.6 KB

Before: 197.8 KB | After: 135.3 KB

Before: 640.0 KB | After: 120.7 KB

After compressing all the images on the page the saving can be seen below.

Click to enlarge

So our total image size saving is 2.4MB – 1.3MB = 1.1MB

A pretty decent saving from just pressing the compress button on 27 homepage images. Also, consider that the user won’t notice any difference in image quality as this method uses lossless compression.


The compression achieved is great for helping us tick off one of the requirements for fast pages with Google. But as we are about to find out Google will likely still complain about two other criteria. When it comes to Google Page Speed insights a page that does not have properly processed images will bring up the following three recommendations:

Here is a break down of how we address each one:

  1. Serve image in next-gen formats – Image formats like JPEG 2000, JPEG XR, and WebP often provide better compression than PNG or JPEG, which means faster downloads and less data consumption. Learn more.
  2. Properly Size images – Your CSS layouts should be responsive and use modern image retrieval techniques that adapt the image size requested based on screen size. Read More
  3. Efficiently encode images – The Tiny PNG integration above will take care of this. This is all about compressing the image to as small as it can get without a visible loss of quality.

So assuming you have already achieved number three using the Tiny PNG integration or another source, let us look at how we can solve the next-gen image requirement.

As a quick side note the testing I did after converting the images to next-gen also ticked item number two above. I don't think this should be relied on however and its best to incorporate responsive images into your projects from the beginning.  

When looking into how to convert images to a next-gen format I opted to target webp. Google has a nice little page explaining the format here.

WebP is natively supported in Google Chrome, Firefox, Edge, the Opera browser, and by many other tools and software libraries.

Once again I opted to look for an API that would provide the conversion for me so that Sitecore could easily connect, send the image and then store the result. All without any extra hosting requirements. I opted to go with Kraken.IO image APIs as they have a free 100MB trial offer and well free is a good price when building proof of concepts. The integration is all available on Aceik’s github repository. Just signup for your own API keys add them to the module settings (in the CMS) and start converting.

To test out just how much this would impact the image payload size for the whole page, I once again converted all the images on the SXA habitat homepage.

Here are the results:

Click to enlarge

So our total image size saving is now 2.4MB – 0.79MB = 1.61MB

The reduction in size from a non-compressed image to a webp formatted image is truly impressive.

A few examples:


Conclusion

I can only conclude by saying that if page speed is really an important factor for your Sitecore project take a look at Tiny PNG. If you want to go next level with your image formats and achieve great compression try out the Kraken.IO API integration as it could be well worth the small subscription fee.


Results

CompressionTotal Image SizeSaving
None2.4 MB
Tiny PNG1.3 MB1.1MB
Kraken.IO (webp)0.79MB1.61MB

Notes:

The module and code mentioned in this blog post are available on Aceik’s Github account. This also contains installations instructions.

GitHub: https://github.com/Aceik/ImageCompression

After installation, your content editors will simply be able to compress and convert images as needed from within the CMS.

Click to enlarge

The Github Readme contains a run down and the standard settings inside Sitecore as shown below:

SXA Speedy – Supercharge your SXA Page Speed Scores in Google

We are excited to preview our latest Open Source module. Before jumping into the actual technical details here are some of the early results we are seeing against the Habitat SXA Demo.


Results:

Results

Before:

After

After:

Before
* Results based on Mobile Lighthouse Audit in chrome. 
* Results are based on a local developer machine. Production results usually incur an additional penalty due to network latency.

Want to know more about our latest open source SXA Sitecore module …. read on ….


I’m continually surprised by the number of new site launches that fail to implement Google recommendations for Page Speed. If you believe what Niel Patel has to say this score is vitally important to SEO and your search ranking. At Aceik it’s one of the key benchmarks we use to measure the projects we launch and the projects we inherit and have to fix.

The main issue is often a fairly low mobile score, desktop tends to be easier to cater for. In particular, pick out any SXA project that you know has launched recently and even with bundling properly turned on its unlikely to get over 70 / 100 (mobile score). The majority we tried came in somewhere around the 50 to 60 out 100 mark.

Getting that page score into the desired zone (which I would suggest is 90+) is not easy but here is a reasonable checklist to get close.

1) Introduce image lazy loading
2) Ensure a cache strategy is in place and verify its working.
3) Dianoga is your friend for image compression
4) Use responsive images (must serve up smaller images sizes for mobile)
5) Introduce Critical CSS and deferred CSS files
7) Javascript is not a page speed friend. Defer Defer Defer

The last two items are the main topics that I believe are the hardest to get right. These are the focus of our new module.

Critical_plus_defer

Check out the GitHub repository.

I have also done an installation and usage video.

So how will the module help you get critical and JS defer right?

Deferred Javascript Load

For Javascript, it uses a deferred loading technique mentioned here. I attempted a few different techniques before finding this blog and the script he provides (closer to the bottom of the article) seems to get the best results.  It essentially incorporates some clever tactics (as mentioned in the article) that defer script load without compromising load order.

I also added in one more technique that I have found useful and that is to use a cookie to detect a first or second-time visitor. Second-time visitors naturally will have all external resources cached locally, so we can, therefore, provide a completely different loading experience on the 2nd pass. It stands to reason that only on the very first-page load we need to provide a deferred experience.

Critical + Deferred CSS Load

For CSS we incorporated the Critical Viewport technique that has been recommended by Google for some time. This technique was mentioned in this previous blog post. Generating the Critical CSS is not something we want to be doing manually and there is an excellent gulp based package that does this for you.

It can require some intervention and tweaking of the Critical CSS once generated, but the Gulp scripts provided in the module do seek to address/automate this.

Our module has a button added into the Configure panel inside the Sitecore CMS. So Content Editors can trigger off the re-generation of the Critical CSS when ever needed.

Generate Critical button added to Configure.

Local vs Production Scores

It’s also important to remember that the scores you achieve via Lighthouse built into Chrome on localhost and your non-public development servers can be vastly different than production. In fact, it’s probably safest to assume that non-production boxes give false positives in the region of 10 to 20 points. So it’s best to assume that your score on production will be a little worse than expected.

Conclusion

It’s a fair statement that you can’t just install the module and expect Page Load to be perfect in under 10 minutes.  Achieving top Page Load Speed’s requires many technical things to work together. By ensuring that the previously mentioned checklists are done (Adequate Servers, Sitecore Cache, Image Loading techniques) you are partway over the line. By introducing the deferred load techniques in the module (as recommended by Google) you should then be a step closer to top score.

For more hints please see the Wiki on Github.

This module has been submitted to the Sitecore Marketplace and is awaiting approval.


Author: Thomas Tyack – Solutions Architect / Sitecore MVP 2019

The 4 most common Sitecore issues we have found our clients are facing

Sitecore is often referred to as the Rolls Royce of CMS platforms, but its power and flexibility come with a caveat – that flexibility can lead to solutions which are functional, but have inherent structural problems. Over time this can lead to anything from slow page performance to the site completely falling over on what should be a small deployment. This can leave you feeling like you’ve paid for a Rolls Royce but received a Ford Pinto.

We here at Aceik perform a lot of work around auditing third party implementations, analysing and remediating these ‘fires’. We are one of, if not the most technically capable Sitecore specialist agencies in Australia, with two out of the total four Australian Sitecore Technology MVPs in our roster and a 100% project success rate across over 50 engagements. As you can imagine we’ve seen all manner of issues.

With this in mind I thought I’d compile some things we’ve often encountered in our travels and send it to you to see if we can help.

1. Not using the license to its full (or even partial) capability

Easily the most common issue we see is implementations which have been developed in such a way that the powerful analytics, personalisation and workflow capabilities of Sitecore are either not turned on, or outright broken.

Compounding the problem is that often the client doesn’t even know what they are missing! The original Sitecore demonstration of these features may have happened in another time long ago with different stakeholders, and the current marketing team is not aware of the capability at their fingertips.

Ultimately what I tell clients is that if all you want is a fast, slick website then you can achieve that in any myriad of technologies. If you are paying for Sitecore – the ‘Rolls Royce of CMS platforms’ you should be getting bang for your buck and capitalising on the ‘platform’. If you look at the image below, you’ll note that the website is only a tiny part of the platform.

analytics

The real value, and what you are ultimately paying for is the ability to track every interaction with your brand, build it into a robust ‘golden record’ of each customer, and then action this information; drive conversions, learn more about your customers, provide them with a better experience and learn where you should spend your marketing dollars.

We guide customers to using their license to the fullest by evaluating where opportunity exists to use Sitecore features, working with them to segment customers and define engagement values and then progressing through a strategy to use that data.

2. Performance Issues

Sitecore features a pretty robust model for caching and squeezing out performance increases – but you have to actually use it! In our experience the caching model Sitecore uses is not well understood by many developers, and encountering a slow website which isn’t actually caching anything is a common occurrence.

My colleague (and Sitecore Technology MVP) Thomas Tyack wrote a great blog on this topic which you can find here.

Outside of this, a lot can be said about developing a site in best practice and understanding the consequences of certain architectural decisions. Something as simple as deciding which method your code will use for retrieving Sitecore items can have dramatic performance costs when users actually start hitting your website. Likewise a deep knowledge of indexing strategies is needed to tune the way search will work to your particular organisation and context.

3. High deployment risks and large cycle times

Another thing we’ve seen time and time again are Sitecore clients that have had so many issues with deployments that what should be an empowering experience – pushing new features to be customer facing, has become a risky, anxious ordeal. One customer hadn’t pushed a feature to their live environment in over a year!

  • Sitecore uses what can be a particularly error prone method of configuration so it’s imperative that deployments have:
    Appropriate Continuous Integration/Continuous Improvement setup that removes the need for anybody to manually log into Sitecore instances and make manual changes. The environments should be ‘what you see is what you get’ setups with no configuration or infrastructure drift between them. This is probably a good time to plug Aceik Express – our best of breed CICD pipeline we endeavor to use on all projects. Each environment is destroyed and recreated from scratch on deployment so everything deployed is tracked and reproducible
  • Business practices in place to empower multiple teams to develop concurrently. This encompasses the structure of the solutions to enable global, reusable components while managing risk around features which have platform wide implications such as Sitecore Pipelines. We also provide guidance as to the best, risk free practices around synchronising content from your live environment back to developers, testing methodology and graceful disaster recovery.

4. Transparency around solution

It’s often said in finance ‘the market can handle good news, it can handle bad news but what it absolutely can’t handle is uncertainty’. So it is with Sitecore development. The platform is so vast, with so many different options for configuration that it is often difficult for clients to have a clear understanding of the state of their implementation. You may be surprised to read this is not even limited to companies without a strong internal Sitecore team. The nature of the platform is such that unless you are developing across many different, varied implementations you may not be across the gotchas and implications around upgrades, integrations and the various technologies which can be used with Sitecore.

For this reason most of our clients engage us with a Technical Account Manager agreement – essentially a pool of hours with a senior architect which the client can use for technical strategic planning around new features, pre-upgrade analysis or just deep dive questions.

Typically we also start our engagements by performing our robust solution audit, a top to bottom investigation of every issue, gotcha and recommendation we have encountered across our cumulative experience with Sitecore. The output of this audit is a prioritised and sized backlog of issues which can blend with your current product backlog and determine dependencies.

Through these two offerings our clients enjoy the clarity to see where they are on their development roadmap, and plan ahead realistically.

I hope this has been food for thought and allowed you to briefly assess where you are on the totem pole of Australian Sitecore implementations. If you think you’re towards the bottom, or even at the top (and want to stay there!) do let us know if we can help.

We have dedicated architects in Melbourne, Brisbane and Sydney and travel interstate frequently to put a friendly face to our Technical Account Managers.

Thanks for your time,

Michal Strzalkowski
Chief Technical Architect – Aceik.

Top 5 Ways to Extend Sitecore HTML Cache

In 2017 I wrote a reasonably long post on all the different considerations a Sitecore Caching Strategy might cover.

Following on from that post its time to share some custom HTML cache extensions that we at Aceik may incorporate into our projects. This count down of custom settings has been collected from across the Sitecore community.

5) Vary By Personalisation

This one (in my opinion) is a must-have if your incorporating personalisation in your homepage.

I admit it will only be effective up to a certain number of content variations and even then needs to be used with caution. Still, if you can save your server and databases from getting hit and help keep Time to First Byte (TTFB) low, its always worth it.

Please note that if you’re displaying a customised message with data that is only relevant to that user, the number of variations may not make it worthwhile.¬† On the other hand, if your showing variations based on a handful of profile card match rules, we found it to be fairly effective.

Code Sample: ApplyVaryByPeronalizedDatasource.cs

Credits: Ahmed Okour

4) Vary By Resolution

Its a fairly common scenario that we want to display a different image based on the users screen size. So it stands to reason that we would need a way to differentiate this when it comes to caching our Renderings.

The particular implementation was used in combination with Scott Mulligan’s Sitecore Adaptive Image Library.

The Adaptive Image library stores the users screen resolution via a cookie in the front end razor/javavscript:

document.cookie = '@Sitecore.Configuration.Settings.GetSetting("resolutionCookieName") =' + Math.max(screen.width, screen.height) + '; path=/';
  • The first time around if no cookie is set it uses the default largest image size as the cache key.
  • If the cookie is set the cache incorporates the screen resolution.
args.CacheKey += "_#resolution:" + AdaptiveMediaProvider.GetScreenResolution();

Code 1:  ApplyVaryByResolution.cs

Code 2: AdaptiveMediaProvider.cs 

Credit:  Dadi Zhao

3) Vary By Timeout

This one’s a little different, it requires not only a new checkbox but also a new “Single-Line text” field that allows you to enter a timeout value.¬† The idea, as you might have guessed, is for the rendering cache to expire after a certain amount of time.

Code 1: ApplyVaryByTimeout.cs

Credit: Dylan Young

2) Vary By Url

An oldy but a goody. I’m a little surprised this one just hasn’t made it into the out of the box product. On the other hand, I can see how it could be overused if you don’t understand the context it applies to. Essentially you can take either the Context Item ID or the raw URL and make your rendering cache vary based on that key.

A good use case for this setting could be for navigation that requires the current page to always be highlighted.

Code 1: ApplyVaryByURLCaching.cs  (Context Item ID formula)

Code 2: ApplyVaryByRawUrlCaching.cs (Raw URL formula)

Credit:  The 10 other people that have blogged about this over the years.

1) Vary By Website

Given Sitecore is an Enterprise content management system we often see multi-site implementations launched on the platform. It makes sense then that you have an option to cache renderings that don’t change all that much on one site but have different content on another.

Example Usage: A global navigation used across all sites that requires some content for the context site to show differently.

Code: ApplyVaryByWebsite.cs

Credit: Younes van Ruth

That rounds out the count down of some of the top ways to extend Sitecore’s out of the box rendering cache. Your renderings will likely use a combination of these settings in order to achieve adequate caching coverage.

For a better idea on how you might add the top 5 above into Sitecore. Please see the technical footnote below. 



 

Technical Footnote:

All these extensions will add an extra checkbox in the Rendering cache tab within Sitecore.

cachesettings

In order for this check box to show up you need to add your custom checkbox fields to the template:

/sitecore/templates/System/Layout/Sections/Caching

You can achieve this in several ways. and there are a lot of other blogs “on the line” that describe how to add in these custom checkboxes so I won’t go into a deep dive here.

With regard to the Helix architecture lets outline one way you could set this up. Aceik has a module in the foundation layer that has all the custom cache checkboxes added to a single template (serialized in Unicorn within that module) . The system template above is then made to inherit from your custom template in order to inherit the custom caching fields.

custom

Sitecore Page Speed: Part 3: Eliminate JS Loading Time

In part 1 & part 2 of our Sitecore page speed blog, we covered off:

  • The Google Page Speed Insights tool.
  • We looked at a node tool called critical that could generate above the fold (critical viewport) CSS code that is minified.
  • We referenced the way in which Google recommends deferring CSS loading.
  • We showed a way to integrate “Above The Fold” CSS into a Helix based project and achieve a page free of render blocking CSS.

In this 3rd part of the series, we will introduce a way to defer the load of all external javascript assets (async).

A reminder that I have committed the sample code for this blog into a fork of the helix habitat example project. You can find the sample here. For a direct comparison of changes made to achieve these page load enhancements, view a side by side comparison here.

Dynamic JS loading Installation Steps:

  1. Inside Sitecore add a new view Rendering that reference the file /Views/Common/Assets/Scripts-3.2.1.cshtml
    • Note down the ID of this rendering and replace in the ID of the rendering in the next step.
  2. Update the Default.cshtml layout to include a new cached rendering.
  3.  @*Scripts Legacy Jquery jquery-3.2.1 *@
     @Html.Sitecore().CachedRendering("{B0DD36CE-EE4A-4D01-9986-7BEF114196DD}", new RenderingCachingSettings { Cacheable = true, CacheKey = cacheKey + "_bottom_scripts" })
    • cacheKey = This variable is something unique that will identify the page. You could use the Sitecore context Item ID or path for example.

Explanation:

The rendering Scripts-3.2.1.cshtml will render out the following javascript onto the page:

var scriptsToLoad = ['//cdnjs.cloudflare.com/ajax/libs/modernizr/2.8.3/modernizr.min.js','//maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js','/assets/js/slick.min.js','/assets/js/global.js','/assets/js/Script.js','/assets-legacy/js/lib/lazyload.min.js'];
src="/assets-legacy/js/lib/jquery-3.2.1.min.js" async defer>
  • First of all, it prints out a JS array of all the scripts that this page requires.
    • This is the array of JS files that comes from Themes and Page Assets inside the CMS. If you are familiar with Habitat Helix this list can be content managed inside the CMS.
  • It then instructs the jquery library to be loaded async (which will not block the network download of the page response).
  • Once jquery is loaded, this modified version of jquery contains some code at the end that will read in the list of scripts dynamically and apply them to the page.
    • This is achieved with fairly simple AJAX load calls to the script URLs.

Outcome:

Once integrated successfully you will end up with a page that does not contain any blocking JS network calls.  The Google Page Speed tool should give you a nice score boost for your achievement in reducing initial load time.


Hints and Tips:

Bootstrapping JQuery Code:

  • jquery Document.Ready() function calls may not fire inside dynamically loaded JS files. This is because the JS file is loaded after DOM is ready and it’s too late for the Document.Ready() event at this stage.
  • As a workaround, you could code your JS files to bootstrap on both the Document.Ready() or whenever $ is not undefined.
  • In the case of dynamic loading in this manner, because¬†jquery was loaded first, $ should not be undefined and your code should be bootstrapped successfully.

Debugging in chrome:

  • When dynamically loading JS files they may strangely not appear in the chrome console debugger as you would normally expect.
  • The workaround for this is to add a comment to the top of each JS library
  • //# sourceURL=global.js
  • This will cause the chrome debugger to list the file in the source tab under the “(no domain)” heading.
  • You will then be able to debug the file as per normal.

Sitecore Page Speed: Part 2: Inlining CSS into Helix

In part 1 of this Sitecore page speed blog, we covered off:

  • The Google Page Speed Insights tool.
  • We looked at a node tool called critical that could generate above the fold (critical viewport) CSS code that is minified.
  • We referenced the way in which Google recommends deferring CSS loading.

In this second part of the Sitecore Page Speed series, I am going to cover off how I would go about achieving this in my Sitecore layout.

Before we dive in I have committed the sample code for this blog into a fork of the helix habitat example project. You can find the sample here. For a direct comparison of changes made to achieve these page load enhancements, view a side by side comparison here.

Installation Steps:

MinimisedCode

  1. For each page that you want to render above the fold CSS we take the minimised code (we generated in the first blog) and put it on the page within the CMS.
  2. Inside the CMS we also create some new renderings in the common project.
    • These renderings are used in the default.cshtml layout.
    • They point to the CSS rendering code.
    • This wrapping technique provides the ability to cache the rendering so that the code in¬†RenderAssetsService.cs does not need to be executed on every single page load.
    • Take note of the IDs of each rendering you will need to copy them over to the CachedRendering IDs shown in step 3 below.
  3. Update the default.cshtml layout with two key renderings.
    • One in the <head> tag that points to /Views/Common/Assets/InlineStyles.cshtml
    •  @*Inline Styles Rendering*@
       @Html.Sitecore().CachedRendering("{B14DA82E-F844-4945-8F31-4577A52861E1}", new RenderingCachingSettings { Cacheable = true, CacheKey = cacheKey + "_critical_styles" })
    • One just before the </body> closing tag that points to¬†/Views/Common/Assets/StylesDeferred.cshtml
    •  @*Styles Rendering Deferred Styles *@
       @Html.Sitecore().CachedRendering("{F04C562A-CBF9-40CF-8CA9-8CE83FDF0BFA}", new RenderingCachingSettings { Cacheable = true, CacheKey = cacheKey + "_bottom_styles" })

StylesDeferred.cshtml  contains logic that will check for inline CSS on every page. If the page contains Inline CSS then the main CSS files will have their network download deferred until later. On the other hand, if the page does not contain any inline CSS then the main CSS files will be loaded as blocking assets. Doing so ensures that the page displays normally in both situations.

  • The¬†cacheKey variable passed to our¬†CachedRendering is simply something to identify the page as unique.¬† You could use the Sitecore context item ID or path for example.

If done correctly you should end up with pages that look normal even with the main CSS files deleted (only do this as a test). The CSS will no longer load via another network that blocks the page and your Google Page Speed rank should recieve a boost.

Sitecore Page Speed: Part 1 : Above The Fold Content

In this series of blogs, I am going to run through ways in which you can increase your score on the Google Page Speed Insights tool.

If you’re not familiar with PageSpeed Insights¬†head over to the page hosted by Google and enter the URL of a Sitecore project you have recently worked on.¬† It will give you a rank out of 100 and then advise on what is wrong with the way your page HTML, JavaScript and CSS assets are loaded. It will also provide feedback on the way your server is setup and how assets are cached.

If you’re getting a score below 50 on Desktop or Mobile I would suggest it’s time to look at ways you can improve your layouts and renderings.¬† A score above 80 and you’re really doing pretty well.

If you got a score of 100 … you should probably be writing this blog instead of me.¬† ūüôā

Topic 1: Above the fold content / Critical CSS

One of Google’s recommendations is to use a technique called:

  • Above the Fold
  • Critical CSS

As you can imagine this refers to the CSS required to render only the visible part of the page.

In order to do this, you render the critical CSS inline within the head tag. The rest of your CSS is then loaded using a deferred script block approach.  This is all demonstrated in a simple example on this page.

The way this works is the inline minified CSS is delivered over the network as part of the page payload. It is not an external asset and will not block the page load via another network request. The page can, therefore, display the visible section (above the fold) immediately after the page is downloaded.

The bulk of CSS required by the page can be loaded a few moments later via deferred network requests.

What is the best way to construct the minimised block of Inline CSS you ask?

Lucky for us a very handy node tool called “critical” is available for download.

You can really easily spin up a gulp script that will generate all the critical CSS for the main pages across your Sitecore website.

A full example of a gulp critical script can be found here.  Download it and simply run:

  • npm install
  • gulp

This will generate a series of CSS files that contain critical viewport CSS.


In the next blog post, we will cover off how to integrate this inline CSS (above the fold) into our Sitecore layouts.


References:

 

Sitecore Azure Search: Top 10 Tips

Its been a while since I first wrote about Azure Search and we have a few more tips and tricks on how to optimise Azure Search implementations.

Before proceeding if you missed our previous posts check out some tools we created for Azure Search Helix setup and Geo-Spatial Searching.

Also, check out the slides from our presentation at last years Melbourne Sitecore User Group.

Ok let us jump into the top 10 tips:

Tip 1) Create custom indexes for targeted searching

The default out of the box indexes will attempt to cover just about everything in your Sitecore databases. They do so to support Sitecore CMS UI searches out of the box.¬† It’s not a problem if you¬†want to use the default indexes (web, master) to search with, however for optimal¬†searches and faster re-indexing time a custom index will help performance.

By stepping back and looking at the different search requirements across the site you can map out your custom indexes and the data that each will require.

Consider also that if the custom indexes need to be used across multiple Feature Helix modules the configuration files and search repositories may need to live in an appropriate Foundation module. More about feature vs foundation can be found here.

Tip 2) Keep your indexes lean

This tip follows on from the first Tip.

Essentially the default Azure Search configuration out of the box will have:

<indexAllFields>true</indexAllFields>

This can include a lot of fields and your probably not going to need every single Sitecore field in order to present the user with meaningful data on the front end interfaces.

The other option is to specify only the fields that you need in your indexes:

<include hint="list:IncludeField"> 
<Text>{A60ACD61-A6DB-4182-8329-C957982CEC74}</Text> 
</include>

The end result will limit the amount of JSON payload that needs to be sent across the network and also the amount of payload that the Sitecore Azure Search Provider needs to process.

Particularly if you are returning thousands of search results you can see what happens when “IndexAllFields” is on via Fiddler.

This screenshot is via a local development machine and Azure Search instance at the Microsoft hosting centre.

Fiddler Index

JSONFIelds

  • So for a single query “IndexAllFields” can result in:
    • 2 MB plus JSON payload size.
    • Document results with all Sitecore metadata included. That could be around 100 fields.

If your query results in Document counts in the thousands obviously the payload will grow rapidly. By reducing the fields in your indexes (removing un-necessary data)  you can speed up query, transfer and processing times and get the data displayed quicker.

Tip 3) Make use of direct azure connections

Sitecore has done a lot of the heavy lifting for you in the Sitecore Azure Search Provider. It’s a bit like a wrapper that does all the hard work for you. In some cases however you may find that writing your own queries that connect via the Azure Search DLL gives you better performance.

Tip 4) Monitor performance via Azure Search Portal

It’s really important to monitor your Azure Search Instance via Azure Portal. This will give you critical clues as to whether your scaling settings are appropriate.

In particular look out for high latency times as this will indicate that your search queries are getting throttled. As a result, you may need to scale up your Azure Search Instance.

In order to monitor your latency times go to:

  1. Login to Azure Portal
  2. Navigate to your Azure Search Instance.
  3. Click on metrics in the left-hand navigation
    • metrics
  4. Select the “Search Latency” checkbox and scan over the last week.
    • graph
  5. You will see some peaks these usually indicate heavy periods of re-indexing. During re-indexing, the Azure Search instance is under heavy load. As long as your peaks under 0.5-second mark your ok.  If you see Search Latency up into the 2-second timeframe you probably need to either adjust how your indexes are used (caching and re-indexing) or scale up to avoid the flow on effects of slow search.

Tip 5) Cache Wrappers

In the code that uses Azure Search, it would be advisable to use cache wrappers around the searches when possible. For your most common searches, this should prevent Azure Search getting hit repeatedly with the same query.

For a full example of cache wrapper checkout the section titled Sitecore.Caching.CustomCache in my previous blog post.

Tip 6) Disable Indexing on CD

This is a hot tip that we got from Sitecore Support when we started to encounter high search latency during re-indexing.

Most likely in your production setup, you will have a single Azure Search instance shared between CM and CD environments.

You need to factor in that CM should be the server that controls the re-indexing (writing) and CD will most likely be the server doing the queries (reading).

Re-indexing is triggered via the event queue and every server subscribes and reacts to these events. Each server with the out of the box search configuration will cause the Azure Search indexes to be updated.  In a shared Azure Search (or SOLR instance) this only needs to be updated by a single server. Each additional re-index is overkill and just doubling up on re-indexing workload.

You can, therefore, adjust the configuration on the CD servers so that it does not cause re-indexing to happen.

The trick is in your index configuration files to use Configuration Roles to specify the indexing strategy on each server.

 <strategies hint="list:AddStrategy">
 <!--
 NOTE: order of these is controls the execution order 
 -->
 <strategy role:require="Standalone OR ContentManagement" ref="contentSearch/indexConfigurations/indexUpdateStrategies/onPublishEndAsync"/>
 <strategy role:require="ContentDelivery" ref="contentSearch/indexConfigurations/indexUpdateStrategies/manual"/>
 </strategies>

Setting the index update strategy to manual on your CD servers will take a big load off your remote indexes.

Particularly if you have multiple CD servers using the same indexes. Each additional CD server would cause additional updates to the index without the above setting.

Tip 7) Rigid Indexes – Have a deployment plan

If your deployment includes additions and changes to the indexes and you need 100% availability of search data, a deployment plan for re-indexing will be required.

Grant chatted about the problem in his post here. To get around this you could consider using the blue / green paradigm during deployments.

  • This would mean having a set blue indexes and a set of green indexes.
  • Using slot swaps for your deployments.
    • One slot points to green in configuration.
    • One slot (production) points to blue in configuration.
  • To save on costs you could decommission the staging slot between deployments.

Tip 8) HttpClient should be a singleton or static

The basic idea here is that you should keep the number of HttpClient instances in your code to an absolute minimum if you want optimal performance.

The Sitecore Azure Search provider actually spins up 2 x HttpClient connections for every single index. This in itself is not ideal and unfortunately, there is not a lot you can do about this code in the core product itself.

In your own connections to other APIs, however, HttpClient SendAsync is perfectly thread safe.

By using HttpClient singletons you stand to gain big in the performance stakes. One great blog article worth reading runs you through the performance benefits. 

It’s also worth noting that in the Azure Search documentation Microsoft themselves say you should treat HttpClient as a singleton.

Tip 9) Monitor your resources

In Azure web apps you have finite resources with your app server plans. Opening multiple connections with HttpClient and not disposing of them properly can have severe consequences.

For instance, we found a bug in the core Sitecore product that was caused by the connection retryer. It held open ports forever whenever we hit out Azure Search plan usage limits.  The result was that we hit outbound open connection limits for sockets and this caused our Sitecore instance to ground to a slow halt.

Sitecore has since resolved the issue mentioned above after a lengthy investigation working alongside the Aceik team. This was tracked under reference number 203909.

To monitor the number of sockets in Azure we found a nice page on the MSDN site.

Tip 10) Make use of OData Expressions

This tip relates strongly to tip 3.  Azure search has some really powerful OData Expressions that you can make use of by a direct connection.  Once you have had a play with direct connections it is surprisingly easy to spin up really fast queries.

Operators include:

  • OrderBy, Filter (by field), Search
  • Logical operators (and, or, not).
  • Comparison expressions (eq, ne, gt, lt, ge, le).
  • any¬†with no parameters. This tests whether a field of type¬†Collection(Edm.String)¬†contains any elements.
  • any¬†and¬†all¬†with limited lambda expression support.
  • Geospatial functions¬†geo.distance¬†and¬†geo.intersects. The¬†geo.distance¬†function returns the distance in kilometres between two points.

See the complete list here.


 

Q&A

Q) Anything on multiple region setups? Or latency considerations?

A) Multi-region setups:¬† ¬†Although I can’t comment from experience the configuration documentation does state that you can specify multiple Azure Search instances using a pipe separator in the connection string.

<add name="cloud.search" connectionString="serviceUrl=https://searchservice1.search.windows.net;apiVersion=2015-02-28;apiKey=AdminKey1|serviceUrl=https://searchservice2.search.windows.net;apiVersion=2015-02-28;apiKey=AdminKey2" /> 

Unfortunately, the documentation does not go into much detail. It simply states that “Sitecore supports a Search service with geo-replicated scenarios” which one would hope means under the hood it has all the smarts to take care of this.

I’m curious about this as well and opened a stack overflow ticket. Let’s see if anyone else in the community can answer this for us.

Search Latency: 

Search latency can be directly improved by adding more replicas via the scaling setting in Azure Portal

replicas

Two replicas should be your starting point for an Azure Search instance to support Sitecore. Once you launch your site you will need to follow the instruction in tip 4 above monitor search latency.¬† If the latency graph is showing consistent spikes and high latency times above 0.5 seconds it’s probably time to add some more replicas.

Dynamic Placeholders and Sitecore Powershell

This post gives a detailed example of a Sitecore PowerShell script that automates the process of inserting content components (renderings) with particular content into a number of items.

Scenario:

Recently a client of ours wanted to roll out a promotion content row to all items of a particular template.

  • These items also happened to be bucket items so there were quite a few of them.
  • The components were usually added to the page via the Experience Editor.
  • The components were used in a building blocks manner:
    • Add a full-width Row
    • Add a Two column component inside the row
    • Add an image in the left-hand column
    • Add a rich text component in the right-hand column
  • The components were added via Dynamic¬†Placeholders.

You could achieve this in certain cases by adding to the Standard Values however if your items already vary drastically from the Standard Values you may not get good enough coverage.

So here is the script with lots of comments so that you can follow along.

View the full script in our GitHub Repo here.

Key Take Aways

  1. We found you need to use Add-Rendering and then Get-Rendering in order to get the correct UniqueID of the rendering just added.
    • In order to construct the Dynamic Placeholder name correctly, you will need the correct UniqueID of the parent just added to the page.
    • The easiest way to facilitate that lookup in Get-Rendering was to give the rendering a unique parameter to identify it.
       (-Parameter @{"InNewPromotionRow"=1;
    • Without this, you may get a list of all renderings that match the non-specific lookup you just used in¬†Get-Rendering
  2. Because of the way Dynamic Placeholders are added to the page this worked best in the Final Layout.
     -FinalLayout
  3. If things go really wrong in your script its best practice to spin up a new Item Version so that you can roll these changes back via another script.
     Add-ItemVersion -Path $i.Paths.Path -IfExist Append -Language "en"
  4. If you have two dynamic placeholders with the same base name then an additional postfix is used to target the second one. So for instance in our case, we had a left and a right-hand column with dynamic placeholders that use the same base placeholder name. In order to target the right-hand side you use “_1” on the end:
     $colWidePlaceholderRight = "/page-body-meeting/row_$rowId/col-wide_$($twoColumnRenderingId)_1"
  5. Nested Dynamic Placeholders: The line of code in point for above also shows how to construct the placeholder key of a nested dynamic placeholder.  In this case, we have a Column inside a Row. The PowerShell script you use to build your components needs to keep track of the placeholder hierarchy so that you can continue to construct nested dynamic placeholders.