Scaling Selenium Testing at Yelp

Ken Struys

  • 4 years at Yelp
  • homepage, signup, search of yelp.com
  • Technical Lead of Webcore (Infra for Web Devs)

What is Selenium?

  • Test Framework For Web Applications
  • Simulate User interactions
  • Code in any language
  • Test every browser

Selenium Server

A Few Terms

  • Selenium RC (2004)
  • Selenium IDE
  • Selenium Webdriver (2009)

Selenium RC

  • Executes JavaScript to drive the browser
  • Still a lot of examples out there
  • Don't use it

Selenium IDE

  • Test Generation
  • Flow Recording
  • Feels really fragile

Selenium Webdriver

  • Replaced Selenium RC in 2009
  • Still uses parts of Selenium RC
  • Addressed a lot of the design issues
  • Object based design - WebDriver + WebElement

Webdriver Examples

Using the Python Library

WebDriver


    def do_selenium_stuff():
            driver = WebDriver(
                command_executor='http://localhost:4444/wd/hub',
                desired_capabilities=DesiredCapabilities.FIREFOX
            )
            driver.implicitly_wait(10)

            driver.get('http://yelp.com')

            for handle in driver.window_handles:
                driver.switch_to_window(handle)

            driver.refresh()
            img = driver.get_screenshot_as_png()
            element = driver.find_element_by_css_selector('.foo .bar')
                            

WebElement


    element = driver.find_element_by_css_selector('.foo .bar')

    element.click()
    element.send_keys('foo bar')

    sub_element = element.find_element_by_id('sub-element')
                        

Selenese API Requests


    POST /session
        desiredCapabilities: {"browser": "firefox", "version": "23.0"}

    POST /session/2c34ac26-2d8c-4bfb-9210-3038f7c92252/element
        using: "css selector"
        value: ".foo input.bar"

    GET /session/2c34ac26-2d8c-4bfb-9210-3038f7c92252/element/0/value
        value: "Text to enter"
                        

Remote Webdriver

  • A Simple Proxy
  • Starts browsers
  • Forwards selenese commands to browsers
  • Browsers plugins manage execution of commands

DEMO!

How we write Selenium Tests at Yelp

Our Workflow

  • Start a sandboxed environment
  • Create fixtures
  • Execute a flow
  • Confirm side effects

Login & Write a Review

Sandboxing

    $ sandbox testify yelp.tests.write_a_review
  • Starts a new mysql instance and creates tables
  • Starts starts apache, memcached, etc
  • Executes command (runs our test)
  • Shutdowns the sandbox

Create Fixtures


    def create_fixtures(email, password):
        user = User.create(email, password)
        business = Business.create()
        return user, business
                    

Create Flows


    def perform_login(driver, user):
        driver.get('http://dev.yelp.com/login')
        driver.find_element_by_id('email').send_keys(user.email)
        driver.find_element_by_id('password').send_keys(user.password)
        driver.find_element_by_css_selector('button[type=submit]').click()

    def perform_write_a_review(driver, rating, review_text):
        driver
            .find_element_by_css_selector(
                'li:nth-child('+ rating +') input[type=radio]')
            .click()
        driver
            .find_element_by_css_selector('input[name=review]')
            .send_keys(review_text)
        driver.find_element_by_id('submit').click()
                    

Create Assertions


    def assert_user_wrote_review(user, business, review_text, rating):
        review = business.reviews[0]
        assert review.text == review_text
        assert review.rating == rating
        assert review.user == user
                    

Write The Test


    def test_write_a_review(driver):
        email = 'kstruys@yelp.com'
        password = 'password'
        review_text = 'This place is awesome!'
        rating = 5
        user, business = create_fixtures(email, password)
        perform_login(driver, user)
        assert_user_logged_in(driver, user)
        driver.get('http://dev.yelp.com/writeareview/' + business.id)
        perform_write_a_review(driver, rating, review_text)
        assert 'Your Review has been posted' in
            driver.find_element_by_id('alert-message')
        assert_user_wrote_review(user, business, review_text, rating)
                    

Selenium Components

API To Widgets


    form = driver.find_element_by_css_selector('.signup-form')

    signup_form = SignupForm(form)
    signup_form.update(
        first_name="Ken",
        last_name="Struys"...
    )
    signup_form.submit()
                        

Page Objects

API To Pages


    class LoginPage {
        public LoginPage(WebDriver driver) {
            this.driver = driver;

            if (!"Login".equals(driver.getTitle())) {
                throw new IllegalStateException("Not the login page");
            }
        }

        public Homepage login(String username, String password) {
            this.fill_form(username, password);
            this.submit();
            return new Homepage(this.driver);
        }
    }
                    

Why is Selenium Important?

Better Coverage

  • Unit testing is necessary but not sufficient
  • Unacceptable to ever break the important flows

More Clientside Code

  • Serverside
    • You control the software/hardware that produces HTML
    • Page refreshes allow flows to restart
  • Clientside
    • Different JS engines performing rendering
    • A minor bug could result in a blank/broken page
    • Logging errors is more difficult

Deployment

  • Dev
  • Stage
  • Prod

Stage Testing is Tedious, Boring and Error Prone

Common Complaints

Selenium is inherently flaky

  • Selenium RC was really flaky
  • Requires education - Dos and Don't
  • Requires Practice!

Ain't nobody got time for that!

  • You're manually testing anyways
  • Integration tests take less time to write
  • It's really not that complicated

Emulate the User

Write tests as a QA employee, not an engineer

JavaScript Injection

  • Avoid it! (as much as possible)
  • Events in javascript are not consistent across browsers/versions
  • More security restrictions
  • Webdriver fires events at the OS Level (when possible)

Adding Selenium To Your Stack

Minimal Setup

  • Every web startup should get basic selenium coverage
  • Run a Docker container for firefox+selenium server
  • Something working in a couple days

Selenium at Yelp

  • ~3-4 Production pushes a day
  • Python Library + Firefox (we want to add more browsers)

4 Years ago

  • Infrastructure setup by an intern (yup, that easy!)
  • Only test major flows - Write a Review, Signup, Login
  • ~ 40 Tests
  • ~ 10-15 minutes to run our selenium suite

Today

  • We learned how to fix and avoid flaky tests (still learning)
  • ~ 1000 selenium tests
  • ~ 30 minutes to run our selenium suite

Flaky Tests

  • Understand flakes vs failures
  • Avoid bad features in selenium (e.g. code injection)
  • Enable implicit wait
  • Selenium RC -> Webdriver

Saucelabs

  • Free for opensource
  • 350 browser/OS combinations version
  • Video Recording/Logging
  • Tunnel to your internal dev environment
  • saucelabs.com

The Future

  • Appium at Yelp
  • Opensource Python Libraries
  • Continuous Testing/Alerting

Selenium Resources

I want to work with passionate people

We're hiring

yelp.com/careers

Questions?