Let’s remind ourselves quickly of why this is important to us. There are many answers to this question. For me- if I had to pick the primary appeal of test automation it would be- *spotting problems early*:

If I introduce a bug in my code - I want to know immediately because if I have to debug it a month from now, it might as well have been written by someone else.

To spell out further how test automation helps with this- if you have good test coverage and you run your tests frequently, you’re likely to spot problems that you’ve introduced *quickly*. If you have a discipline of running your tests every time you check in your code, that is called 'continuous integration'.

Other motivations for Test Automation include: you find manual testing tedious, you don't have a QA team, you want to try 'Test Driven Development', etc.

That’s the motivational portion over with - let’s confront a hard truth. Test automation is not for everyone. Let’s make sure, we are on the same page about some key things.

Test automation is hard:

• 1. it takes discipline to keep at it
• 2. creative discipline to apply and design
• 3. intellectual discipline to interpret and act upon

I’ve decided to divide our topic into 3 sections: (1)PL/SQL Testing, (2) Browser Testing and (3) Performance Testing - this corresponds to the order in which I think Oracle APEX organizations should prioritize Test Automation implementation.

Let’s start with PL/SQL testing. By my analysis, this will probably give you the biggest return on effort. The 1st piece of good news here, is that this isn’t a crowded space - there are a few options that you may recognize but without question the most popular and successful choice is utPLSQL - a software originally conceived of by Steven Feuerstein in the 90s.

Demo

Here's a gif of the demo I provided in my presentation. Notice how human-readable the output is.

The Challenge of PL/SQL Testing

What about procedures/functions:

• 1. With no output parameters?
• 2. With non-deterministic output?
• 3. That aren’t easily repeatable?

The above are some common examples of hurdles you will have to confront as you implement PL/SQL testing. There are no easy solutions to any of them. But recognize that an imperfect/partial test is better than no test at all.

PL/SQL Testing will change how you code

You will write smaller procedures/functions to isolate the code that is difficult to test.

Tip : Recognize that testing is about approximation and compromise.

This may be the most popular test automation focus. Let’s take a moment to review what is a very crowded competitive landscape (above). I have only looked into a handful of the above tools so I am unqualified to make across the board recommendations. I’ll share my browser automation journey - my very 1st introduction to any kind of automated testing was with Selenium, which was and continues to be an extremely popular solution. However, I never loved Selenium and when I joined Fabe, I took it as an opportunity to give Cypress a try and I am very impressed. I do not miss Selenium one bit.

Selenium vs Cypress

The above screen is from my presentation in which I show that 59 lines of python code to use Selenium translates to just 11 lines in Cypress code.

Selenium Demo

A very simple demo from my presentation of what Selenium can do.

Cypress Demo

The same demo but this time in Cypress. Notice how the left-hand nav bar gives a friendly description of steps taken.

The challenge of Browser Testing

• 1. ‘User story’ (‘end-to-end’) tests can take 10min+ and are therefore expensive
• 2. The risk of ‘flake’ increases multiplicatively with every incremental HTTP request in a given test
• 3. Most of the things we care about testing take a lot of set-up (authentication, authorization, seed data)

As a result of these challenges, browser tests are ungainly, unreliable and rarely run.

There's no completely avoiding these hurdles. The best recommendation I have for making your peace with them is embracing the approach of testing 'features' in addition to 'stories'. Feature tests / Unit tests can be short and therefore minimize time and set-up.

Browser testing will change how you code

You will:

• 1. Label and apply your elements more thoughtfully (no more reusing css classes with misleading names)
• 2. Move more of the logic into the database so utPLSQL can test it
• 3. Make features more easily isolated (eg. relax some security in your dev environment)
• 4. Learn the value of ‘Unit testing’

Last stop 'Performance testing' - a fairly crowded space but not as crowded as that of browser testing. I have limited familiarity with most of these softwares. I can point out that Apache Jmeter has been an industry standard for 20 year and will probably continue to be so. In my opinion, there has only been modest innovation since then.

Let's define some terms. We have 3 principal types of performance testing:

• 1. load/reliability : observe the health metrics of your system at given load of users
• 2. stress torture test : extreme test until system failure
• 3. endurance /continuous test : observe the health metrics of your system as you run your test for days on end

Jmeter Demo

A very simple demo from my presentation. Jmeter requires that we translate what we do in the browser into a series of HTTP requests.

The challenge of Performance Testing

Load testing’s challenge is two-fold but really they feed into each other:

• 1. A resource challenge- The system you care about is your Production environment. You do not want to run your performance test against this environment because of the risk that it will impact customers. You therefore have to approximate your system in a test environment in the hope that when you simulate 50,000 users against your test system, it will be a good approximation of what would happen if that same volume hit your production system. Additionally, it takes real server power to simulate 50k users of activity. So all together, we're taking a lot of server power and some of production grade - so it can get very expensive.
• 2. Interpretation challenge: relatedly - let's say your performance test is showing that wait times for your customers begins to sky rocket after 50k users. This sounds important. But it's not always easy to have confidence that the fault isn't with the test environment.

In summary - you have one side saying 'I can't use these results because the system is unreliable', and the other side saying 'I can't afford to improve the system, it's too expensive.'.

The compromise approach requires you to start your performance testing before you think you need it. You take your perfomance test at a point of being reasonably content with performance against your test environment but you don't scrutinize the meaning of the health metrics. However, you can then use those metrics as the base line to all future tests. If performance dips or soar against this base line, you will therefore be confident it's the code and not the system.

Some callouts:

• 1. Selenium can be written in 9 different languages (Python,Java,C#,Ruby,Javascript,Groovy,Perl,Scala & PHP) so it is not a self-evident thing to assess the difficulty of writing or reading Selenium tests. I have a comfort level with Jquery that makes Cypress fairly accessible to me.
• 2. I have installed Selenium 3+ times and it takes me over an hour every time because I forget all the dependencies, so I've given Selenium the maximum score for difficulty for installation.
• 3. Cypress gets a better score for 'Trust' relative to Selenium because it is more intelligent about waiting for resources to load before running its test. Selenium tests, consequently, have a higher incidence of 'false positives'.
• 4. Jmeter and Gatling get high difficulty scores for 'Act on Output' because it is often not self-evident whether the Performance issue is related to the software or the hardware.
• 5. Jmeter gets a high difficulty score for 'Run' because it's ability to simulate volumes of users is constrained by the processing power of the machine running the test. Enterprise-level performance tests often employ

• Get started with utPLSQL
• Learn about Selenium
• Get started with Cypress
• Get started with Apache Jmeter
• Learn about Gatling