Performance Resources
Search
Syndicate
Poll
Navigation
Recent Customer engagements – handling file uploads and streaming content
Our Professional Services team has been extremely busy over the past several months helping Customers prepare for what proved to be a very busy online shopping season. I’d like to share with you a couple of unique challenges that we came across and how we were able to use the BrowserMob platform and some special customizations in order to successfully test what normally would be impossible using just Selenium or a traditional http-based testing approach.
File Uploads
We worked with a large online greeting card company to completely automate the creation of a card, complete with multiple file uploads. Typically, we see file uploads implemented using a standard web form which are easy to automate with Selenium and the BrowserMob filePath directive. However, in this case, the upload used a Flash component to control the upload which then presented a native OS dialog to the user to select the file as in the screenshot below.
Unfortunately, these types of dialogs are not accessible by Selenium and thus require native mouse and keyboard integration as mentioned in this this earlier blog post. As the interaction was relatively simple (just select the file to be uploaded), we were able to automate this using BrowserMob’s special native keyboard and mouse utilities.
As a result, we were able to completely automate the creation of a card from start to finish and the Customer was subsequently able to integrate this script into their regular testing process. Please keep in mind that this type of interaction is quite brittle as it relies on specific x,y coordinates to be clicked on and can cause the script to break over time as objects are re-positioned on the site. In this case, as the interaction was relatively simple, it worked out well. For more complex interactions using Flex, we strongly recommend checking out Gorilla Logic’s solutions as mentioned in this article.
Streaming Media
Another Customer came to us with a requirement to simulate a streaming media event for hundreds of users they were hosting using a Windows Media infrastructure. We were able to customize the boot-up process of our standard load testing agents and install the Mplayer plugin for Firefox which allowed us to stream the content across hundreds of browser and multiple locations. The Customer was then able to diagnose performance issues and bottlenecks in advance of the actual live event, saving them a lot of time and effort in the process.
During this engagement, we also used a special statistical sampling technique to verify that the content was playing as expected. In the script, we purposely threw an error, approximately 5% of the time which then took a screenshot and video of the transaction which we were able to review in real-time along with the Customer’s team. The relevant script snippet to accomplish this is included below:
function randomid(minVal,maxVal) { var randVal = minVal+(Math.random()*(1+maxVal-minVal)); return Math.floor(randVal); } if (!browserMob.isValidation()) { var id = randomid(1,1000000000); //randomly take a screenshot 5% of the time var shot = randomid(1,20); if (shot == 5) { throw("Catch body broken: IOException " + "user id " + userid + "-" + id + " " + new Date().toString()); }What type of special testing requirements do you have that are complex and challenging? Contact us and let us know, as we can help you come up with a solution!
Tip & Trick: Schedule your load test for up to 60 minutes
Consider scheduling your load tests for up to 60 minutes. You can always stop the test earlier. By scheduling it for 60 minutes, you retain the option to run the test longer or even pause/resume the test, effectively getting a second test for free. The cost of a shorter test, i.e. 5, 10, or 15 minutes, is the same as a 60 minute test, so there’s really nothing to loose.
Repost: Straight From the Horse’s Mouth: Load Testing Best Practices from Our Customers
As you’ve likely seen, we pulled together a comprehensive list of load testing best practices for you to visit, and revisit, at your convenience to help you maintain high performance websites and applications that keep your customers happy.
As helpful as we hope these guidelines are, we know sometimes our readers want to hear how load testing works from the end users themselves. We’ve compiled a few best practices that have come right out of the mouths of customers using our load testing services:
Test different types of traffic at different times of the day.
“We started by using a model of expected user behavior—consumers, manufacturers, and retailers—and testing features individually for each type of traffic. We see higher traffic at certain times of the day, so that was a factor too. We tested lots and lots of scenarios and finally, to be really sure, ran a ‘perfect storm’ scenario with traffic spikes from all our users.” – Dave Cumberland, VP of engineering, Shopatron
Test when you expect hikes in traffic.
“We had a good handle on the size and scope of the stress tests to run… It took a couple of days to build the scripts and subsequently each test iteration was completed within a single day. We were able to demonstrate that the platform could easily handle the expected load over the holiday season after just one iteration. We plan to execute additional iterations after significant code changes or in anticipation of events that will impact system load.” – Steve Romney, CTO, Shutl
Replicate the true user experience.
““First, we tested what it’s like when people log in [to our online auction site]. Then we replicated heavy bidding, with bids coming every 15 seconds. Finally, they tested the refresh process, where bidders get real-time updates. This allowed us to pinpoint what to expect during peak loads.” – Brian Ellis, Director of IT, Taylor and Martin
For more information on how we’ve helped our customers and what they have to say about our load testing services, take a look at our case studies here.
Do you have any best practices to share from your own load testing experiences?
Firefox 7 support
We are happy to announce that we recently added support for Firefox 7 to our monitoring and load testing products! You can now explicitly specify the Firefox version your script will run against. Currently we support both Firefox 3.6 and Firefox 7.
For example, the following script will open the Firefox 7 browser and navigate to a url:
var selenium = browserMob.openBrowser("FF7"); browserMob.beginTransaction(); browserMob.beginStep("Home Page"); selenium.open("http://seleniumhq.org/"); browserMob.endStep(); browserMob.endTransaction();The same is possible when using the Selenium 2.0 API (WebDriver):
var webDriver = browserMob.openBrowserWebDriver("FF7"); browserMob.beginTransaction(); browserMob.beginStep("Home Page"); webDriver.get("http://seleniumhq.org/"); browserMob.endStep(); browserMob.endTransaction(); Firefox 3.6Existing monitors and load tests will continue to use Firefox 3 by default. New monitors will default to using Firefox 7, so you if you want to create a new monitor and have it run against Firefox 3.6, specify “FF3″ as the browser type:
var selenium = browserMob.openBrowser("FF3");Future Versions
In the future, we will add support for more Firefox versions. If you want to keep up with the latest version of Firefox installed on our systems, you will need to specify “FF”:
var selenium = browserMob.openBrowser("FF");When we upgrade to a later version of Firefox, the script will automatically use the new version.
Local ValidatorLocal validator will use the Firefox version you have installed by default. You can override the locations it looks in for the different browsers in a property file. The file is located in your home directory on Unix and under your user folder on Windows.
$HOME/.browserMob/config.properties
For example, if you have Firefox 7 installed in the default path and Firefox 3 under a different directory, you can use the following config.properties to point local validator to the Firefox executables:
FF3 = C:/Program Files/Mozilla Firefox 3/firefox.exe FF7 = C:/Program Files/Mozilla Firefox/firefox.exeScripting for BrowserMob using WebDriver
Hello Readers!! It gives me immense pleasure in meeting you with yet another article on WebDriver. WebDriver has been creating a lot of buzz among the Selenium community lately and BrowserMob, an active member of the Selenium community, wasted no time in announcing its support for the WebDriver API.
I hope you had the chance to read the great introduction article on the Webdriver API by fellow blogger Ben.
In this article I would like to touch on the basics of scripting with the WebDriver and the BrowserMob API.
I’m sure many of you, who are new to this API are asking yourselves these questions just like I did.
- What is WebDriver?
- Why is it needed?
WebDriver is a framework for automated testing of web applications. WebDriver is a part of Selenium 2.0, which is a major departure from the 1.0 model. Selenium 1.0 is a client-server architecture which uses the RC libraries to program tests that communicate with the server, and the server relays those commands to a browser.
The primary downside of the 1.0 model is that it is written in JavaScript. Browsers impose a strict security model on any JavaScript that they execute in order to protect a user from malicious scripts. Additionally, the API for RC has grown over time that users find it hard to keep up with this ever growing dictionary of methods. The WebDriver helps overcome both these hurdles with an object oriented API that controls the browser itself rather than running as a JavaScript application within the browser.
WebDriver has five major Interfaces – ImeHandler, Navigation, Options, TargetLocator , Timeouts, the methods of which fall into following three categories.
- Control of the browser itself
Eg: get(), navigate(), close()etc
- Selection of WebElements
Eg: findElement(), findElements() etc
- Debugging aids
Eg: getCurrentUrl(), getPageSource(), getTitle() etc
Now that we have a clear understanding of what WebDriver is and what its benefits are, let’s take a deep dive into the various useful WebDriver methods exposed via BrowserMob through some example scripts. Let’s look at this simple script below, to begin with.
var driver = browserMob.openBrowserWebDriver(); var tx = browserMob.beginTransaction(); var step = browserMob.beginStep("Home Page"); driver.get("http://seleniumhq.org/docs/06_test_design_considerations.html"); // polls DOM every 500 ms for ten seconds until the text 'Test Design Considerations' is found driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS); // Content validation - find element by xpath driver.findElement(By.xpath("//div[@id='test-design-considerations']/h1[contains(text(), 'Test Design Considerations')]")); browserMob.endStep(); browserMob.endTransaction();Deciphering the script…
- var driver = browserMob.openBrowserWebDriver();
The openBrowserWebDriver()method of WebDriver starts a WebDriver browser session. - driver.get(“http://seleniumhq.org/docs/06_test_design_considerations.html”);
The ‘get’ method which is equivalent to selenium.open() is used to navigate to any web page. - driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
driver.findElement(By.xpath(“//div[@id='test-design-considerations']/h1[contains(text(), 'Test Design Considerations')]“));WebDriver has a blocking API. i.e. WebDriver will wait until the page has fully loaded (that is, the “onload” event has fired) before returning control to your test or script. However, under some conditions it is possible for a ‘get’ call to return before the page has finished loading. The classic example is JavaScript starting to run after the page has loaded (triggered by onload). Browsers (e.g. Firefox) will notify WebDriver when the basic HTML content has been loaded, which is when WebDriver returns. It’s difficult to know when JavaScript has finished executing, since JS code may schedule functions to be called in the future, depend on server response, etc. WebDriver cannot wait for all conditions to be met before the test proceeds because it does not know them. The solution to ensure whether the element involved in the next interaction is present and ready is to use the ‘Wait’ class to wait for a specific element to appear. This class simply calls findElement over and over, until the element is found (or a timeout has expired). Since this is the behavior desired by default for many users, a mechanism for implicitly-waiting for elements to appear has been implemented. This is accessible through the Timeouts() interface of the WebDriver class (WebDriver.manage().timeouts() ). In the example script above, the driver.manage.timeouts().implicitlyWait call polls the DOM every 500 ms for 10 seconds until the text ‘Test Design Considerations’ is found.
Tips:
- Optionally you can reset the Implicit Timeout to zero using the code below, as once set, it remains effective until the life of the WebDriver instance. The default setting is zero, i.e zero wait time. driver.manage().timeouts().implicitlyWait(0, TimeUnit.SECONDS);
- It is possible to set a timeout for the entire step as such using the setStepTimeout method of the browserMob API. // Setting a step timeout of 10 seconds browserMob.setStepTimeout(10000); browserMob.beginStep(); ……. browserMob.endStep();
- Alternatively, browserMob’s waitFor() method could be used in place of the implicitlyWait() for content validation. Modifying our script with waitFor()… var step = browserMob.beginStep("Home Page"); driver.get("http://seleniumhq.org/docs/06_test_design_considerations.html"); browserMob.waitFor(function() { return check = driver.findElement(By.xpath("//div[@id='test-design-considerations']/h1")).getText().toLowerCase().startsWith("test design considerations"); }, 10000); if (!check) throw "Content validation failed"; browserMob.endStep();
Great!! We just wrote our first WebDriver script. We now know how to interact with page elements via the findElement method of WebDriver’s WebElement Interface using XPath. But does WebDriver provide other alternatives for interaction? Absolutely. Below are few alternative ways. As implied through the script, using the ‘findElement’ is a great way to validate page content before proceeding to the next step in the script.
By.cssSelector:
driver.findElement(By.cssSelector(“div#test-design-considerations > h1″));
By.linkText:
driver.findElement(By.linkText(“Data Driven Testing”));
By.partialLinkText:
driver.findElement(By.partialLinkText(“Data Driven”));
By.id:
driver.findElement(By.id(“side”));
By.className:
driver.findElement(By.className(“expandable”));
By.name:
driver.findElement(By.name(“q”));
By.tagName:
driver.findElement(By.tagName(“input”));
driver.findElements:
In situations where you have to iterate through multiple elements with the same id/class/tag name etc or have to pick one randomly, the ‘findElements’ method comes in handy. The example script below uses this method to pick a random suggestion from an AutoSuggest list.
As you can see, I’ve also managed to introduce a few other useful methods through this script. The comments above each one of them describe their purpose.
Tips:
- sendKeys can also be used to simulate pressing of the keyboard keys by using the “Keys” class. It is possible to call sendKeys on any element. The code below triggers the “ENTER’ key press to force a form submission. The list of Keys supported can be found here. driver.findElement(By.id('finalExpenses')).sendKeys(Keys.RETURN);
- With advanced locators, it is possible to traverse down the DOM with a single line of code as in below:
driver.findElements(By.tagName("ul")).get(1).findElements(By.tagName("li")).get(2).click();
As the index starts with zero, the line of code above would click on the third “li” of the second “ul” on the page.
A few more common page element interactions in detail…
How to handle frames?
Swinging between frames is easy with the switchTo() method of the TargetLocator interface of the WebDriver class.
- Switch to the frame from main window
- By index driver.switchTo().frame(0);
- By frame name driver.switchTo().frame("frameName");
- Switch back to the parent window driver.switchTo().defaultContent();
The following example illustrates a simple swinging between two frames on a web page. Let’s assume the page has two frames, frame1 and frame2, out of which frame1 has a link and frame2 has a form.
driver.switchTo().frame("frame1"); driver.findElement(By.linkText("SPAZ")).click(); driver.switchTo().defaultContent(); driver.switchTo().frame("frame2"); driver.findElement(By.id("field_0")).sendKeys("anu"); driver.findElement(By.className("form_button")).submit(); driver.switchTo().defaultContent();Try running this snippet commenting out the “driver.switchTo().defaultContent();” before switching to frame2 and you would end up with an error that frame2 cannot be located. This explains the need for switching back to the parent page from inside a frame first, before navigating to another frame on the same page.
How to handle JavaScript Alert and Confirmation boxes?
Confirmation boxes are handled the same way as the Alerts.
- Switch focus to the alert/confirmation box var alert = driver.switchTo().alert();
- Dismiss – equivalent to hitting “Cancel” button on the Confirmation box or ignoring an Alert by closing it. alert.dismiss();
- Alternatively accept – equivalent to hitting “OK” on the Alert or the Confirmation box alert.accept();
- Handle Confirmation boxes the JavaScript way driver.executeScript("window.confirm = function(msg){return true;};");
How to handle Pop-up Windows?
browserMob.beginStep("Handle Popup Windows"); driver.get("http://www.javascript-coder.com/files/window-popup/javascript-window-open-example1.html"); // Get the handle to the current window var currentHandle = driver.getWindowHandle(); driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS); // Open Pop-up driver.findElement(By.partialLinkText("Open the JavaScript Window")).click(); // Switch focus to Pop-up using its name driver.switchTo().window('mywindow'); // Alternate way - if the Pop-up window doesn't have a name, which is very common - // get the handle to the last opened window var handles = driver.getWindowHandles().toArray(); var id = handles.length - 1 driver.switchTo().window(handles[id]); //close popup driver.executeScript("self.close ();"); //switch focus to parent window driver.switchTo().window(currentHandle); browserMob.endStep();How to interact with SELECT drop-downs?
var lastItem = driver.findElement(By.xpath("//select[@class='dropDown']/option[last()]")); lastItem.click();How to simulate advanced interaction such as Drag and Drop?
The Actions subclass of the Selenium Interactions class makes emulation of these complex user gestures easier.
- Drag an element to drop on to another element driver.get("http://devfiles.myopera.com/articles/735/example.html"); var from = driver.findElement(By.xpath("//ul[@id='Fav']/li[3]")); var to = driver.findElement(By.xpath("//ul[@id='Tol']/li")); Actions(driver).dragAndDrop(from,to).build().perform();
- Drag an element to an offset driver.get("http://jqueryui.com/demos/draggable/"); var draggable = browser.findElement(By.id("draggable")); Actions(driver).dragAndDropBy(draggable, 200, 10).build().perform();
I hope this article served as a comprehensive tutorial, covering most of the basics of WebDriver, if not all. See you soon with more useful WebDriver tips and tricks. Happy testing until then!!!
Repost: Website Monitoring Is Just as Important as Analyzing Traffic Data
A LinkedIn user recently posed the question: “What are the best practices for monitoring, collecting and analyzing traffic data for a large content site?” Ian White, lead engineer at Neustar, jumped into to tell readers why active monitoring is just as important as analytics. Website monitoring ensures your site is functioning properly anytime from anywhere.
Below we’ve outlined a few of his key points that will help you monitor your site so you can sleep well at night knowing your website is always up and running.
Monitor everything. You need to be alerted whenever a critical system process goes offline. This includes web servers, cache, servers, databases, etc. Understand that you may not know everything you should monitor until it breaks for the first time, but that only serves as lessons learned for additional metrics to monitor and make for a more failsafe website over time.
How you monitor matters. Whether the purpose of your website is to log in a user or add an item to a shopping cart, your customers depend on each click of the mouse to be fully functional. These interactions require monitoring that automates real browser interactions—like sending headers, setting cookies or executing JavaScript—and alerts you if something unexpected occurs. This provides a realistic view of the customer experience.
Monitor from multiple locations. To ensure your website is fully functional from every coordinate on the globe, you need to monitor your site from different geographic locations. There is no “magic number” for how many locations you should monitor from, but we recommend at least three. For example, if your company is based in the U.S., we suggest you monitor from two locations in the States and at least one in Europe or Asia.
Implement Real User Monitoring (RUM). In addition to monitoring your website uptime, you should also test the load times for your website visitors. Similar to Google Analytics, RUM involves placing a JavaScript beacon on your site. Every time a user visits your site, the beacon records the load time and you are alerted whenever this time goes above a specified threshold.
Establish tiered alerting. If a tree falls down in a forest, does it make a sound? The same thought process behind this popular riddle goes into a tiered alerting system. If the individual responsible for fixing a downtime issue doesn’t check his or her email at 3:00am regularly, with a simple email alert system he or she will not hear if/when your site goes offline. Services like PagerDuty ensure that alerts run up the chain of command with continual forms of alerts until someone responds via email, text message or phone call.
For more information on Ian’s response or to join in on the discussion, click here.
Repost: Testing a Flex client with BrowserMob
Thanks to Dave Thompson for sharing his experience with using BrowserMob to test the performance of a Flex app. To read the full article, check out it here.
How to Keep Up With the Holiday Rush: 8 Tips for Retailers Looking to Conquer Spikes in Website Traffic
It’s almost that time of year again. Holiday shoppers looking to stay at arm’s length from the seasonal mall madness will undoubtedly turn to the Internet to make their gift purchases. In fact, ShopperTrak predicts national retail sales will rise 3% during November and December this year as compared to the same time period last year.
In order to keep up with this expected spike in website traffic, there are a few steps ecommerce companies should take to ready their site. In order to help, we’ve rounded up the top eight load testing and website monitoring tips to help ecommerce sites stay on their “A-game” this holiday season.
1. Start load testing now. Procrastination can be a worrisome habit. Put off today what you can do tomorrow, and you may find your ecommerce site has crashed. According to IBM report, Act, Don’t React: A Proactive Business Continuity Solution Protects your Revenues and Reputation, the average revenue loss per hour of downtime is $1.01 million—a price tag your business cannot afford.
2. Set objectives. Look at industry trends and establish what is “acceptable performance” for your organization. Example Key Performance Indicators (KPI’s) before load testing can include:
- 90% of pages should load in 4 seconds or less
- All business transactions should take less than 1 minute to complete
- Category search should take no more than 6 seconds to complete
3. Use real browsers. Thanks to cloud computing, it is now cost effective to load test and monitor a website using real browsers. Real browsers let you create more life-like site traffic, unlike virtual browsers that simply mimic a browser’s http request / response sequence. You’ll come away with a clearer view of your end user’s experience.
4. Test beyond the firewall. The most frequently visited parts of your site are the most important to test: applications supporting your home page, browsing, product selection, checkout and exit. Without external testing, you’re only getting half the picture.
5. Revisit frequency: Consider increasing your monitoring frequency to make sure nothing goes unnoticed. You need to be notified immediately if there is any change in your website’s behavior.
6. Collaborate. Advertising and special promotions will affect traffic to your website. Coordinate with your marketing department to ensure your website can handle traffic upticks caused by marketing campaigns.
7. Document. Establish a holiday preparation process and document all of your load testing and monitoring activities. That way you’ll have all the records you need to perform a post-mortem and improve for next year.
8. Choose the Solution that Works for Best You. When working with load testing providers, you typically have two choices: (a) on-demand services that let you run tests 24/7 or (b) full-service testing, which normally includes a dedicated engineer. If you have the resources and expertise on staff, and require greater flexibility for when you test, on-demand is a good choice (Tip: See if you can get started with a free trial.) If you could use additional expertise with testing and analysis, including professional recommendations and reporting, then full-service load testing is the way to go.
Visit http://www.browsermob.com for more information about our on-demand load testing solutions or http://www.webmetrics.com for more information on our performance monitoring and full-service load testing solutions.
Come one, come all! Announcing BrowserMob First Friday Feedback Office Hours
Looking for a way to keep up on website performance best practices, load testing tips and all things related to the success of your websites and applications? Then join us for our new BrowserMob office hours.
We’re pleased to announce that starting October 7, SMEs from our Engineering, Professional Services, Customer Service and Sales teams are at your service for one hour on the first Friday of every month. Feel free to pick their brains and ask anything that’s on your mind. They’re happy to share success stories and lessons learned from their work with other customers.
Find full details on the new monthly tradition below:
What: First Friday Feedback, the official BrowserMob office hours
When: 9:00 – 10:00 a.m. PST, The first Friday of every month, beginning October 7
Where: Via WebEx (details to come once you register below)
Who: A diverse team of experts from various BrowserMob teams
How: Register here.
If you’d like to shoot our SMEs a question or feedback ahead of time so that we can be sure to cover it on Friday, please send us an email. We look forward to seeing you there!
Monitoring outage from 1:01 to 3:01 PDT
From 1:01 am to 3:01 am PDT on Sept 18, 2011 the data center that houses our secondary queue server became inaccessible from all monitoring locations. During the outage we were not running any monitoring jobs. At this time everything is running normally and our network ops team is working to determine the root cause.
Normally an outage of one of our queueing clusters would not be an issue, as we have both automatic and manual failover policies in place. But this time we were in the middle of migrating the physical server of our primary queue server and were unable to switch over. There are a few spots we have already identified where we can improve our infrastructure to prevent any further outages like this.
Browsermob unreachable from multiple locations
For 42 minutes tonight, 21:02 EDT to 21:44 EDT, Browsermob was unavailable from multiple locations. We worked quickly to resolve the issue as soon as it was detected and are happy to announce that service has been restored. If you have any questions for us, please let us know at support@browsermob.com.
Monitoring unavailable for 37 minutes
From 9:25pm PDT through 10:02pm PDT tonight, monitoring was unavailable. An investigation is being performed right now to determine the root cause of this isolated issue. We are very sorry for any inconvenience you may have experienced, and will learn from this incident to prevent this in the future. If you have any questions, please contact us at support@browsermob.com.