About testing in CI/CD


#1

Hi.
I’ve read lots of articles about CI/CD (including every article in the Katalon blog) and I still don’t understand when do you actually write tests.
I mean, they say that in CI you commit code to the shared repository, then the build is created, and unit-tests starts to take place, then integration tests, system and if everything is ok - you can deploy it to a production. They say that ALL this CI/CD cycle must take no more than 10 minutes, otherwise there is no use in CI/CD if it is slow.
So I think I can understand about unit-tests (the developer can make this itself), regression tests (they are the same usually). But how do you do integration/system and other testing with new features? The developer has just written the code - and in 10 minutes the system would be completely tested. How so? You don’t know how and what exactly the developer is going to write. Aren’t you going to have a lot of time to actually think, write the test cases and then test all of it? I mean isn’t it takes more than 10 minutes?


#2

Easily more than ten minutes. I don’t know what information you were reading, but it surely doesn’t apply to most automation test suites.


#3

they smoke good stuff


#4

Now, to detail a bit:

Up to now looks ok-ish, this process may take less than 10 mins (but here is a long discussion, it depends on the complexity of the application)
But this is just the time from when the developer pushes the change until the build is done. Does not count also the time needed to actually write tests (and here, mostly unit test, code coverage, linting and so on are counted)

you can deploy it to a production.

sure, if you are dumb. usually, after the build passes, it starts the fun (the QA engineer job).
Various functional tests: smoke, regressions, integration’s and so on. That process can take a lot (depends if the tests are already written for the new features, the amount of changes, if the builds pass or fail and has to be investigated blah blah)
Usually this is done in a separate qa environment (sometime can pass through more, e.g dev > qa > staging).
Only after the application is qualified > deploy to production and run the tests for the px environment (at least a smoke)


#5

found a good introductory article on this subject (altough there are always shades of grey):


#6

What im going to say is all really based on your application but you can sort of get the idea -

Ideally if you are working in an “agile” environment then you should have a period of refinement where developers and a member of QA go through what the project is exactly. From there as QA you should ideally receive a set of User stories or Use cases. From these as a tester you can start to create Tests to meet these requirements. You may do these in an automation type notation such as Gherkin to speed up the process when you receive a build. Obviously there is still going to need to be a period of time where you can actually create the methods behind the tests and map out the new pages. Depending on complexity, this can take a few hours or couple days etc.

Now is this regarding to execution in the pipeline? because then 10 minutes is achievable by running multiple workers in headless etc.

If they are regarding to the process of having tests written and it fully tested in 10 minutes, then like Ibus has said, i want whatever they are smoking.

The real benefit of automation is in the long run, obviously there is some time invested when you receive the build to create the tests but once they are created, the benefit is being able to run multiple times and not take up resources when doing so.

Really i think you just need to clarify with whoever has given you these figures, what they are expecting from automation


#7

yeah … exactly, ideally. everybody want’s to be agile … but nobody writes clear specs :smiley:
not to mention also that the specs are changing all the time based on the development approach … or the analyst dreams … or the phase of the moon.

i worked for a while with ‘scrum wannabe’ environment (where in theory the developers starts coding shoulder-to-shoulder with the qa engineer developing tests and they meet at the end of the sprint and they drink a beer and everybody is happy)
The only thing scrum related was … this is how the daily meetings were named :smiley:
(and the beer part … because f … this sprint, the weekend is coming)


#8

and yeah, this is also a good point.

  • hey boss, we need more nodes to parrarelize the builds blah blah…
  • denied, we pay the cloud provider on an per/instance quota, re-use the existing infrastructure … :blush:

#9

We’re stretching the topic now, but this is “General Discussions” so I guess we’re not straying too far.

The history of “agile” (at least as originally proposed and defined) is an exercise in wishful thinking. When newer (or just “other”) ideas came along, they were “wedged in” and promoted as “belonging” under the agile umbrella - because people want to say they’re “doing agile”.

That’s not engineering or science. There’s no discovery here, only evolving ideas. It’s gray, not black and white. It’s inexact science, not exacting.

In 10 years or less, “agile” will be unrecognizable or forgotten.

Fact (based on my long experience in the field). Large companies can afford ten meetings discussing one topic. Small companies hold one meeting and discuss ten topics.

Which is best? Answer: Neither. They both “work” and function to serve their purposes.

Science (including engineering) discovers laws. Laws stand the test of time. Agile is not a set of laws. It does not contain any laws. It will not stand the test of time. So don’t put all your money on it. Something else will “appear” soon enough (and be compared to it, positively and negatively at first).

Summary: A real scrum :slight_smile:

image


#10

@Russ_Thomas love your example!
yah … in the end is a ballance between ‘what we want’ and ‘how to do it’.
long time ago, if i remember it right, an japanese guy wrote a book named ‘wishful thinking’.
i realy forgot the name of the author … but i am positive is a book any of us may like to read

le: from the history i read, also the agile concept came from clever japanese guy (not kazurayam altough he is clever too)


#11

I meant to add, one of the Agile Manifesto authors is one of my heroes: Martin Fowler.

Also, one of the original authors has in the last year, decried the current “popular” understanding of the original manifesto’s aims. I’ve forgotten who and, after searching long and hard, I failed to find a link.

He was quite specific that what most teams are doing was never intended as part of the philosophy. Some parts actually run counter to the ideal.

Suffice to say, he made my points far better than I did.


#12

I also am a fan of they way you have put it Russ, Just a shame you couldn’t pick some better teams to show the example of a Scrum :wink:


#13

Be careful Harry… you’re on thin ice there. :angry:


#14

as long as is not Potter is ok … :))

@harry i consider the example most relevant for this topic. the team know what to do. the managers … hard to say :stuck_out_tongue:


#15

Good practices are tests should be established as much as possible and coverage should be the same as well. However, this takes so much time to achieve this milestone. CD/CI is born to test as FAST as possible to provide FAST feedbacks for involved team members if we exclude CI pre-test stages such as cleanup, setup, etc…whatever

That being said, establish tests on CI/CD practices require a test strategy as well such as if the test suites we are going to use take a long time (time varies depends on your project in this context) then just schedule to run it nightly. If unit tests only, then it should always require to be executed for every changeset. Time is an important factor to control distributed, and as Katalon article refers it’s just a common referral in most projects. Some projects can accept tests run longer than 10 minutes DEPENDS on which kind of test suites are executed

My experiences are if you feel it takes very long to run a simple test, then you have to optimize your tests, not just to blame for CI/CI tools