The increasing adoption of mobile applications as part of a company’s online services leads to the question whether we need to monitor like other parts of our IT infrastructure. As they are part of our shipped application services we need to ensure they are working properly. However, not every application must be monitored the same [...]

One of the big topics in the BigData community is Map/Reduce. There are a lot of good blogs that explain what Map/Reduce does and how it works logically, so I won’t repeat it (look here, here and here for a few). Very few of them however explain the technical flow of things, which I at [...]

dynaTrace AJAX Edition 3.4 now gives you full JavaScript, AJAX, Network and Rendering analysis support for Mozilla Firefox 8 and 9 as well as Internet Explorer 8 and 9. If you want to optimize the performance of all your users’ browser versions, have a look at the dynaTrace AJAX Edition Premium to guarantee the best [...]

Obviously, cloud computing is not just a fancy trend anymore. Quite a few SaaS offerings are already built on platforms like Windows Azure. Others use Amazon’s EC2 to host their complete infrastructure or at least use it for additional resources to handle peak load or do “number-crunching”. Many also end up with a hybrid approach [...]

Today’s web sites are often cluttered up with third party content that slows down page load and rendering times, hampering user experience. In my first blog post, I’ve presented how third party content impacts your website’s performance and identified common problems with its integration. Today I want to share the experience I have made as [...]

Most production monitoring systems I have seen have one major problem: There are too many JVM’s, CLRs and Hosts to monitor. One of our bigger Customers (and a Fortune 500 Company) mastered the challenge by concentrating on what really matters: The Applications! Ensure Health The following dashboard is taken directly from the production environment of [...]

Some time back I planned to publish a series about java memory problems. It took me longer than originally planned, but here is the second installment. In the first part I talked about the different causes for memory leaks, but memory leaks are by far not the only issue around java memory management. Edit: A [...]

Today is great day for us. Due to our continuously growing user base we have decided to migrate our blog to a bigger server. As a performance blog we strive to provide great performance to our readers. Our performance monitoring has shown us that our end user response times were not that great so we [...]

Like everybody else it took me a while to wrap my head around the BigTable concepts in Cassandra. The brain needs some time to accept that a column in Cassandra is really not the same as a column in our beloved RDBMS. After that I wrote the first Web Application and run into a pretty [...]

MMT 28 was a great conference. A lot of great talks and a very interested audience. I was lucky to speak about Web performance. As promised you can find my slides below.  Also feel free to post questions on my talk. Web Performance Optimzation View more presentations from Alois Reitbauer.

When load testing an application, the first set of tests should focus on measuring the maximum throughput. This is especially true of multi-user, interactive applications like web applications. The maximum throughput is best measured by running a few emulated users with zero think time. This means that each emulated user sends a request, receives a response and immediately loops back to send the next request. Although this is artificial, it is the best way to quickly determine the maximum performance of the server infrastructure.
<h2>Little’s Law</h2>
Once you have that throughput (say X), we can use Little’s Law to estimate the number of real simultaneous users that the application can support. In simple terms, Little’s Law states that :

N = X  / λ

where N is the number of concurrent users,  λ is the average arrival rate and X is the throughput. Note that the arrival rate is the inverse of the inter-arrival time i.e. the time between requests.

To understand this better, let’s take a concrete example from some tests I ran on a basic PHP script deployed in an apache server. The maximum throughput obtained was 2011.763 requests/sec with an average response time of 6.737 ms, an average think time of 0.003 secs when running 20 users. The arrival rate is the inverse of the inter-arrival time which is the sum of the response time and think time. In this case, X is .2011.763 and λ is 1/(0.006737 + 0.003). Therefore,

N = X / λ  = 2011.763 * 0.009737 = 19.5885

This is pretty close to the actual number of emulated users which is 20.

 

Estimating Concurrent Users

This is all well and good, but how does this help us in estimating the number of real concurrent users (with non-zero think time) that the system can support ? Using the same example as above, let us assume that if this were a real application, the average inter-arrival time is 5 seconds. Using Little’s Law, we can now compute N as :

N = X /λ  = 2011.763 *  5 = 10058 users.

In other words, this application running on this same infrastructure can support more than 10,000 concurrent users with an inter-arrival time of 5 seconds.

What does this say for think times ? If we assume that the application (and infrastructure) will continue to perform in the same manner as the number of connected users increase (i.e it maintains the average response time of 0.006737 seconds), the the average think time is 4.993 seconds. If the response time degrades as load goes up (which is usually the case after a certain point), then the number of users supported will also correspondingly decrease.

A well-designed application can scale linearly to support 10′s or 100′s of thousands of users. In the case of large websites like Facebook , Ebay and Flickr, the applications scale to handle millions of users. But obviously, these companies have invested tremendously to ensure that their applications and software infrastructure can scale.

Little’s Law can be used to estimate the maximum number of concurrent users that your application can support. As such, it is a handy tool to get a quick, rough idea. For example, if Little’s Law indicates that the application can only support 10,000 users but your target is really 20,000 users, you know you have work to do to improve basic performance.

While we've touched upon client side caching in our series on Web performance, we haven't discussed how client caching has grown more rich and useful over the years. In the initial days of the Web and the HTTP/1.0 protocol, caching was mostly limited to a handful of headers, including Expires, If-Modified-Since, and Pragma: no-cache. Since then, client caching has evolved to embrace greater granularity. Some new technologies even permit the deployment of offline-aware, browser-based applications.

Browser Request Caching

The most common and oldest type of client-side caching on the client is browser request caching. Built into the HTTP protocol standard, browser request caching allows the server to control how often the browser requests new copies of files from the server. We discussed the major aspects of browser request caching in part 1 of our series. Over time, Webmasters have taken to using different headers to improve caching on their site, including:

Pragma: no-cache. This old directive is used mostly by HTTP/1.0 servers, and instructs a client that a specific response's contents should never be cached. It is used for highly dynamic content that is apt to change from request to request.

Expires. Supported since HTTP/1.0, this header specifies an explicit expiration date for cached content. It can be superseded by the value of the Cache-Control header. For example, if Cache-Control: no-cache is sent in a response, this will take precedence over any value of the Expires header.

If-Modified-Since: Since the HTTP/1.0 protocol, clients have been able to use this header to request that the server only send data if the resource has been changed since the specified date. If there have been no changed, the server returns an HTTP 304 Not Modified response.

Last-Modified. This HTTP/1.0 and 1.1 header designates when the resource was most recently changed. Browsers usually supply this value as the value of the If-Modified-Since header.

Cache-Control. This core directive, introduced in the HTTP/1.1 standard, specifies whether a response's contents can be cached, and if so, for how long. The header "Cache-Control: no-cache" obsoletes the "Pragma: no-cache" header of the HTTP/1.0 protocol.

ETag. The ETag ("entity tag") header is a hash value that is specific to a given version of a resource. It can be used by the client in conjunction with the If-Match, If-None-Match, and If-Range headers to decide whether it should generate a new request for the latest version of a resource. The format of entity tags themselves is defined in section 3.11 of RFC2616.

Note that this header and the Last-Modified header are exclusive; servers should set one or the other. The ETag header is new with the HTTP/1.1 protocol standard.

For modern applications, the good folks at Google recommend setting one of either Cache-Control or Expires, and one of either Last-Modified or ETag.

 

Javascript/AJAX Caching

With the advent of JavaScript and AJAX, more Web applications are downloading data dynamically. JavaScript developers can use the XmlHttpRequest object to fetch data in XML (or other) format, and display it in real time without forcing a refresh of the entire page. This presents opportunities for finer-grained caching based on the nature of the data displayed within the page.

AJAX applications can still use all of the browser request caching mechanisms discussed above. The resource requested by the XmlHttpRequest object will be stored in the browser's file cache just as other HTTP objects are. A given AJAX application can go further and make refresh calls to the XmlHttpRequest object using programmatic rules. In his article "An AJAX Caching Strategy", Bruce Perry shows how he uses a custom CacheDecider object that he wrote in JavaScript to determine when to update an AJAX display of oil, gasoline, and propane prices.

 

HTML5 Caching

Developers creating HTML5 applications can create fully offline-aware applications using the HTML5 ApplicationCache interface. The Application Cache uses a cache manifest file to specify which files in an HTML5 application can be used offline, and which files require a network connection. The manifest may also specify a list of fallback files for network resources when the user is offline. For example, instead of fetching the file /get-data.php when disconnected, the manifest can instruct the browser to display the file /offline.html instead. This manifest is referenced in the HTML element of an HTML5 app:

<html manifest="manifest.appcache">
...
</html>

 

Conclusion

Web performance optimization is very important, and today's Web application development team can boost site performance and improve its site's load testing scores by selecting from a variety of client side caching techniques. An effective client side caching strategy can reduce load times by several factors. The most recent innovations in client side caching, such as the HTML5 Application Cache, enable an application to run (though perhaps in a more limited form) even without a network connection present.

Identifying Memory Leaks due to object references in Finalize queue


Load the SOS debugger extension for a CLR 4.0 application

.loadby sos clr

1) Identify the objects in finalizeQueue which survived Garbage collection

!fq

SyncBlocks to be cleaned up: 0

MTA Interfaces to be released: 0

STA Interfaces to be released: 0

----------------------------------

generation 0 has 1 finalizable objects (000000008ba63058->000000008ba63060)

generation 1 has 15 finalizable objects (000000008ba62fe0->000000008ba63058)

generation 2 has 14884 finalizable objects (000000008ba45ec0->000000008ba62fe0)

Ready for finalization 0 objects (000000008ba63060->000000008ba63060)

…

000007ff01d58610 2 1024 System.Data.DataTable

000007ff0164e298 4 1120 System.Diagnostics.Process

000007ff01d82738 8 1728 System.Data.DataColumn

000007ff017235a0 16 1920 System.Threading.OverlappedData

000007ff001f1780 28 2464 System.Threading.Thread

000007ff006764b8 52 3744 System.Reflection.Emit.DynamicResolver

000007ff0026fba8 66 4224 System.Threading.ReaderWriterLock

000007ff01e7bb00 314 10048 System.Data.SqlClient.SNIPacket

000007ff01e717e0 546 21840 System.Data.SqlClient.SNIHandle

000007ff01e49f30 314 32656 System.Data.SqlClient.SqlConnection

000007ff02933bf0 302 45904 System.Data.SqlClient.SqlDataAdapter

000007ff0166d240 398 70048 System.Diagnostics.PerformanceCounter

000007ff01e0d0e8 1510 338240 System.Data.SqlClient.SqlCommand

000007ff0057d8f8 11209 358688 System.WeakReference



2) You can also print the details of the finalizable objects for Gen2 using the above details

dd 000000008ba45ec0 000000008ba62fe0


3) Identify the suspected object having leaks

!dumpheap -type System.Data.SqlClient.SqlCommand

Address MT Size

0000000010c43f40 000007ff01e0d0e8 224

0000000010c44020 000007ff01e0d0e8 224

0000000010c44198 000007ff01e0d0e8 224

0000000010c44278 000007ff01e0d0e8 224

0000000010c444e8 000007ff01e0d0e8 224


4) Identify the GC roots for the objects. It contains the call stack

!gcroot 0000000010c43f40

Scan Thread 15 OSTHread 1bc0

Scan Thread 16 OSTHread 1d74

Scan Thread 19 OSTHread 3200

Scan Thread 17 OSTHread 21f0

Scan Thread 18 OSTHread 3564

Scan Thread 20 OSTHread 322c

Scan Thread 21 OSTHread 2b80

Scan Thread 28 OSTHread 2e5c

Scan Thread 29 OSTHread 35d0

Scan Thread 30 OSTHread 3bc

Scan Thread 31 OSTHread 2770

Scan Thread 33 OSTHread 19b4

Scan Thread 34 OSTHread 534

Scan Thread 35 OSTHread 3280

Scan Thread 36 OSTHread 1908

Scan Thread 37 OSTHread 344c

Scan Thread 38 OSTHread 13d4

Scan Thread 39 OSTHread 21d4

Scan Thread 40 OSTHread 31a4

DOMAIN(0000000001AB88B0):HANDLE(Pinned):1217c0:Root: 000000002070f040(System.Object[])->

00000000108c9ac0(System.Collections.Hashtable+SyncHashtable)->

00000000108c91e8(System.Collections.Hashtable)->

000000001171efc0(System.Collections.Hashtable+bucket[])->

00000000120a3768(System.Collections.Hashtable)->

00000000120a37c0(System.Collections.Hashtable+bucket[])->

00000000120a3820(System.Collections.Hashtable)->

00000000120fa180(System.Collections.Hashtable+bucket[])->

00000000120b7200(System.Collections.Generic.Dictionary`2[[ATOM.AS.CobolBase.CobolProgramName, ATOM.AS.CobolBase],[ATOM.AS.CobolBase.CobolProgram, ATOM.AS.CobolBase]])->

00000000120b7360(System.Collections.Generic.Dictionary`2+Entry[[ATOM.AS.CobolBase.CobolProgramName, ATOM.AS.CobolBase],[ATOM.AS.CobolBase.CobolProgram, ATOM.AS.CobolBase]][])->

00000000120b7258(ATOM.AS.CobolBase.CobolProgram)->

00000000120b7310(System.Collections.Generic.List`1[[ATOM.AS.CobolBase.IProgramEvents, ATOM.AS.CobolBase]])->

0000000019b0f918(System.Object[])->

0000000019b0f590(ATOM.AS.DataAccess.DataAccess)->

0000000019b13c40(ATOM.AS.DataAccess.ProviderSQL)->

0000000019b13d70(System.Data.SqlClient.SqlCommand)->

0000000019b15978(System.Data.SqlClient.SqlCommand+CachedAsyncState)




@2011, copyright Vamsidhar Tokala

There are `python-matplotlib` and `python-numpy` packages in the Debian/Ubuntu repos.

However, if you want to run in a virtualenv (with no-site-packages), and pip install these packages from PyPI, you need some system dependencies installed first to build with:

$ sudo apt-get install build-essential python-dev libfreetype6-dev libpng-dev python-virtualenv

Then, you can create a virtualenv, and the installers for Numpy and Matplotlib will work:

$ virtualenv env $ cd env $ source bin/activate (env)$ pip install numpy matplotlib ... ... Successfully installed numpy matplotlib Cleaning up...

(tested on Ubuntu Oneiric 11.10 and Ubuntu Precise 12.04 alpha)

Last time I talked about the key differences between RDBMS and the most important NoSQL databases. The key reasons why NoSQL databases can scale the way they do is that they shard based on the entity. The „simplest“ form of NoSQL database shows this best, the distributed Key/Value Store. Last week I had the chance [...]

The busiest online shopping time in history has brought a significant increase in visits and revenue over last year. But not only were shoppers out there hunting for the best deals. Web performance experts and the media were out there waiting for problems to happen in order to blog and write about the business impact [...]

Happy New Year! I mean that literally...it is a happy new year for web performance geeks like you and me.

Did you ever read The Hobbit? There was a clever exchange between Gandalf and Bilbo where the meaning of "Good Morning" was discussed. Are you wishing me to have a good morning or are you saying it's a good morning whether I want it or not?

Well, 2012 is going to be a great year for the web performance and load testing industry. I wish that you have a happy new year, AND it will be a happy web performance new year whether you want it or not.

Load Testing for Holiday Season was Awesome

LoadStorm begins 2012 with an upbeat outlook because the last few months of 2011 showed over a 400% increase in load testing volume.

The actual number of load tests being executed by our customers were maybe only double the number in 2010, but the scale of tests was much bigger. It is also interesting to note that many more online retailers were running tests of 25,000+ concurrent users. We had many calls with traditional brick and mortar companies that were putting significant investment into their web store capabilities - including speed and scalability.

Anecdotally, we can share with you that it was a good investment because some of those customers told us that their online sales had risen as much as tenfold (10x) over the previous year! That's great news for all of us web performance engineers.

Not only does it seem clear to us that web stores are getting high priority for e-commerce, but it is also clear that web performance has gotten much more attention from the C-suite. Several of our customers mentioned their load testing projects were being driven from executives worried about site crashing under heavy traffic. That's outstanding! Finally, the stories of Web site performance failures is getting the attention it deserves. I guess it was tough to ignore all the headlines blasting companies like Target when their site crashed in 2011.

Operations and marketing leaders are starting to understand the correlation between web performance tuning and profitability. Web sites are not just online brochures, nor are they just a secondary revenue channel. The message that is coming through loud and clear is that faster sites make more money.

In more than one project-related conference call with companies running load tests of 50,000-100,000 concurrent users, there was a VP of Marketing being very active in driving her/his team regarding the results he/she was expecting. It was refreshing (somewhat shocking) to hear a 60 year old traditional advertising agency veteran telling everyone on the call that "sub-second response time is imperative to success!" I loved it. The web coders...not so much because they had lots of optimization ahead of them.

Online Sales Heading North

The trends are exciting. The attention and high priority placed on web performance is logical because the stakes are so high. More money is flowing every year from buyers to web stores.

People were spending well this holiday season and Cyber Monday 2011 experienced double-digit growth over 2010. Some key activity metrics from IBM's Benchmark December holiday report :

• Department stores online sales were up 18 percent over December 2010.
• Total online sales were up 7.5 percent over 2010.
• Mobile device buyer traffic jumped from 5.6% of all sessions to 14.6 %. That's 160% higher than last year.
• Sales from mobile devices doubled, reaching 11% versus 5.5% in December 2010.
• Christmas Day 2011 online shopping grew by 16.4% over 2010 - people kept on shopping.

Conclusion - Web Performance is Hot

Money is the lifeblood of business. ROI drives business decisions. Better Web performance results in higher sales.

Web performance engineers rejoice! We are now more important than ever to the CEO. Job security, promotions, raises, and lots of other goodness will follow swiftly!

The busiest online shopping season is about to start and it’s time to make a quick check on whether the top shopping sites are prepared for the big consumer rush or whether they are likely going to fail because their pages are not adhering to Web performance best practices. The holiday rush seems to start [...]

In our past installments on Web performance optimization, we've seen how caching, server configuration, and the use of Content Delivery Networks (CDNs) can increase a Web site's responsiveness and improve Web performance metrics. Most of the techniques we've reviewed have focused on configuring the Web server or optimizing server applications. Unfortunately, a Web page that downloads quickly but is slow to parse or execute on the client will appear just as slow to a user as if the Web server were on its last megabyte of memory. In this article, we'll discuss some ways that Web page content can be streamlined for an optimal client-side experience.

 

Streamline JavaScript Includes

JavaScript abounds on the Web. From jQuery to Dojo, the Web is full of JavaScript libraries that can easily be dropped into a Web application. And any site whose developers are actively adding features is going to accrue its own storehouse of .js files. Unless a site's JavaScript is carefully managed, its Web pages could end up making a dozen or more separate requests for scripts. As we've already discussed in our article on
web performance optimization non-caching strategies, the more requests your site makes, the slower it will load.

Tip of the hat to TechAttitude.com for the graphic showing how Web page sizes and number of objects have grown tremendously over the past 16 years. In the interesting article they state, "...the average size of a web page has increased by more than five times since 2003" and "the use of mulitmedia is increasing by 100% each year".

Follow these guidelines to manage and reduce the burden of JavaScript on your Web pages.

• Use Minify to combine script files. As discussed previously, Minify will compile all of your disparate JavaScript files safely into a single .js file. This can collapse the number of HTTP requests made by a page from a half-dozen or more down to one single request.
• Remove unnecessary or expensive includes. JavaScript files whose functionality is no longer used should be stripped out. Site owners should also consider removing functionality that causes a script to load and execute multiple times on a page (e.g., a social media script that forces a script fetch for every individual post in a blog scroll).
• Consider a JavaScript compressor. If your site is loaded with client-side code, it may be worthwhile to use one of the many freely available JavaScript compressors on the market to shrink down your files. JavaScript Compressors remove extraneous whitespace, strip out comments, and shorten variable names. This means that JavaScript files not only faster to download, but faster to interpret and execute. A good JavaScript compressor is ShrinkSafe, which is also built into the dojo JavaScript toolkit.

 

Performance Tune Your JavaScript

Of course, optimizing JavaScript includes won't help much if the code you use is inefficient. Poorly written or maintained JavaScript can harm browser performance in a number of ways. Consider:

• Closures. Closures are the most common cause of memory leaks in JS. A closure is a powerful mechanism that allows functions to be treated as first-class objects by retaining their execution context. Often, hard to spot circular references between a closure and one of its scoped variables causes the closure to remain in memory indefinitely. While the performance impact may be negligible at first, it builds quickly over time, until the user's browser is running so poorly that she has no choice but to restart it.
• Circular references. Two objects that hold references to one another will never be garbage collected by the JavaScript run-time engine, resulting in even more leaked memory.
• DOM interactions. JavaScript is at its slowest when interacting with the Document Object Model (DOM).

Developers can surmount these problems using certain tools and techniques:

• Performance test JavaScript code. Performance testing can be as simple as using timestamps to measure the execution time of a particular segment of code. Developers who require a more detailed analysis of their code may opt to use a performance testing package such as jsPerf or jsLitmus.
• Avoid using expando properties. Another common source of memory leaks. As Gregory Baker and Erik Arvidsson note in their article on JavaScript performance, "You can use expando properties without introducing memory leaks, but it is pretty easy to introduce one by accident."
• Avoid closures whenever possible. The IE team has provided an in-depth explanation of why developers should avoid closures except when absolutely necessary.

The book Speed Up Your Site: Web Site Optimization goes into even greater detail on subjects such as minimizing DOM interaction, selecting efficient algorithms, and improving loop execution.

Another in-depth resource that I recommend you explore is JavaScript Performance Rocks! by Amy Hoy & Thomas Fuchs. It's a bundle of PDF books, demos, and a DOM Monster profiling tool for $39.

 

Remove Unnecessary or Expensive HTML

Finally, developers can squeeze even more performance out of their downloads and page rendering by cleaning up extraneous markup in HTML documents. This includes, but is not limited to:

• Reducing the use of DOM elements, such as unnecessary DIV and SPAN elements. These are expensive to create and maintain, and can cause memory bloat over the long term.
• Remove unnecessary IFRAME elements.
• Remove any IMG or IFRAME tags with a blank src attribute. These cause an additional request to be sent to the server.
• Remove extraneous comments. They increase download time, and are stripped out by the parser anyway.
• Keep META tags and META content to a minimum. Other than the Description tag, most of the META elements no longer have any value for SEO.

 

Conclusion

The greatest factors in Web site performance are the number of requests between the client and the server and the speed of the individual requests. It is best to address these issues first through caching, tuning the Web application (database interactions), and Web server configuration before tackling HTML optimization. Development teams and Web site operators who have already addressed these issues can then optimize their client-side code to ensure that their pages load and execute efficiently once they are delivered to the client.

"SST (selenium-simple-test) is a framework built on Selenium WebDriver, using Python to make writing functional web tests easier with code."

Since early 2011, I have been working for Canonical on the Infrastructure Systems Development team (Core Dev Ops).

[pic of canonical-isd-hackers at UDS-P-Orlando]

A by-product of our recent development efforts is a web testing framework. It has been available on Launchpad for a while, but I've never really announced it in public. We are using SST internally, and I want to expose it to a wider audience.

What is Selenium WebDriver?

Selenium (WebDriver) is a popular open-source library for automating browsers. It can be used to create functional/acceptance tests of a web application. The Selenium client bindings provide API's that allow you to programatically drive a browser and access web content/elements. The bindings are available and supported for many languages and platforms.

While working directly with Selenium API's from code is fine for ad-hoc browser interaction, it is rather low-level and lacks things necessary for creating suites of automated web tests. For larger-scale testing, you will soon want to use a framework to help organize, execute, and report.

Introducing SST...

SST aims to keep things simple.

Tests are made up of scripts, created by composing actions that drive a browser and assert conditions. You have the flexibility of the full Python language, along with a convenient set of functions to simplify web testing.

SST framework consists of:

• user actions and assertions (API) in Python
• test case loader (generates/compiles scripts to unittest cases)
• console test runner
• data parameterization/injection
• selectable output reports
• selectable browsers
• headless (xvfb) mode
• screenshots on errors

Test output can be displayed to the console, saved as an HTML report, or JUnit-compatible XML for compatibility with CI systems.

SST is free open source software (Apache Licensed). SST is primarily being developed on Linux, specifically Ubuntu. It should work fine on other platforms, but any issues (or even better - patches) should be reported on the Launchpad project:

• https://launchpad.net/selenium-simple-test

I just uploaded SST 0.1.0 to PyPI:

• http://pypi.python.org/pypi/sst

go ahead, give it a try:

• `$ [sudo] pip install sst`

documentation and more info:

• http://testutils.org/sst

a sample test script in SST:

from sst.actions import * go_to('http://www.ubuntu.com/') assert_title_contains('Ubuntu homepage')

Here is the development progress of SST (shown as a code_swarm visualization) over the past 8 months:

Special thanks to all the SST code committers so far:

• Danny Tamez
• Julien Funk
• Kenneth Koontz
• Leo Arias
• Lukasz Czyzykowkski
• Rick McBride
• Sidnei da Silva

Extra special thanks to SST's initial creator:

• Michael Foord

Happy New Year!

-Corey Goldberg