[asterisk-dev] Branching in the Testsuite

[Kevin Harwell]

Greetings,

We're thinking about adding a branching system to the Asterisk Testsuite.
Each branch would be named the same as, and correspond to, an Asterisk
branch. So for instance the following branches would probably be created:

13, 14, 15

For each release of Asterisk we will also create a tag in the Testsuite
that corresponds to that release's tag. That way someone could checkout
both tags for easy testing

Other advantages? Most all, if not all, the current versioning stuff found
in the Testsuite could go away, or be safely ignored moving forward. The
versioning has become a bit cumbersome especially when you have to make a
backward incompatible change to a test. Moving the version control out of
the Testsuite and into a version control system should alleviate the need
for this moving forward.

Please let us know your thoughts and considerations on moving forward with
this model. Especially any potential pitfalls or problems you might see
with it.

Thanks!

[Corey Farrell]

It would be nice to strip out some/most of the per version conditionals
in tests.  The need to always cherry-pick changes is the only pitfall I
see with your proposal.  If a test never had per version differences
then the cherry-pick is trivial amount of extra work, but for tests with
differences per branch it would mean dealing with conflicts.  In the
testsuite I think I'd rather deal with per branch conditionals over
merge with conflicts.

I just want to propose an alternative way of (mostly) accomplishing the
goal.  We could create a tag '12' from current master.  This would
represent the last revision of the testsuite known to work with Asterisk
12 and below.  We would then be free to remove compatibility with EOL
Asterisk branches from testsuite master.  We would tag testsuite '14'
soon after September 26th, 2018 (EOL for Asterisk 14).  This would avoid
multiplying the number of gerrit reviews for testsuite changes, but it
would require continuing to maintain version declarations for supported
versions of Asterisk.  I think this would be a good trade-off so we
aren't stuck with all the 1.8/11 baggage.

Post by Kevin Harwell
Greetings,
We're thinking about adding a branching system to the Asterisk
Testsuite. Each branch would be named the same as, and correspond to,
an Asterisk branch. So for instance the following branches would
13, 14, 15
For each release of Asterisk we will also create a tag in the
Testsuite that corresponds to that release's tag. That way someone
could checkout both tags for easy testing
Other advantages? Most all, if not all, the current versioning stuff
found in the Testsuite could go away, or be safely ignored moving
forward. The versioning has become a bit cumbersome especially when
you have to make a backward incompatible change to a test. Moving the
version control out of the Testsuite and into a version control system
should alleviate the need for this moving forward.
Please let us know your thoughts and considerations on moving forward
with this model. Especially any potential pitfalls or problems you
might see with it.
Thanks!
--
Kevin Harwell
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at:http://digium.com &http://asterisk.org

[George Joseph]

It would be nice to strip out some/most of the per version conditionals in
tests. The need to always cherry-pick changes is the only pitfall I see
with your proposal. If a test never had per version differences then the
cherry-pick is trivial amount of extra work, but for tests with differences
per branch it would mean dealing with conflicts. In the testsuite I think
I'd rather deal with per branch conditionals over merge with conflicts.

Given the low commit volume, even lower commit volume against existing
tests, high compartmentalization and small nature of the files in the
testsuite, I don't see conflicts being a big thing. If there are any,
then they're probably justified.

I just want to propose an alternative way of (mostly) accomplishing the
goal. We could create a tag '12' from current master. This would
represent the last revision of the testsuite known to work with Asterisk 12
and below. We would then be free to remove compatibility with EOL Asterisk
branches from testsuite master. We would tag testsuite '14' soon after
September 26th, 2018 (EOL for Asterisk 14). This would avoid multiplying
the number of gerrit reviews for testsuite changes, but it would require
continuing to maintain version declarations for supported versions of
Asterisk. I think this would be a good trade-off so we aren't stuck with
all the 1.8/11 baggage.

We were thinking of taking the current master, copying it to 13, 14, 15,
and "legacy" then starting with the next release our release scripts would
automatically create tags in the testsuite just as they do in asterisk.

This also makes Jenkins much happier BTW.

Greetings,
We're thinking about adding a branching system to the Asterisk Testsuite.
Each branch would be named the same as, and correspond to, an Asterisk
13, 14, 15
For each release of Asterisk we will also create a tag in the Testsuite
that corresponds to that release's tag. That way someone could checkout
both tags for easy testing
Other advantages? Most all, if not all, the current versioning stuff found
in the Testsuite could go away, or be safely ignored moving forward. The
versioning has become a bit cumbersome especially when you have to make a
backward incompatible change to a test. Moving the version control out of
the Testsuite and into a version control system should alleviate the need
for this moving forward.
Please let us know your thoughts and considerations on moving forward with
this model. Especially any potential pitfalls or problems you might see
with it.
Thanks!
--
Kevin Harwell
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at: http://digium.com & http://asterisk.org
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
asterisk-dev mailing list
http://lists.digium.com/mailman/listinfo/asterisk-dev

[Corey Farrell]

Sounds good.  So after this is done am I correct that when my review
against asterisk 13 is approved jenkins will 'gate' using the 13 branch
of testsuite?  Will the testsuite also get minor branches (like 13.19)?

Post by Corey Farrell
It would be nice to strip out some/most of the per version
conditionals in tests.  The need to always cherry-pick changes is
the only pitfall I see with your proposal.  If a test never had
per version differences then the cherry-pick is trivial amount of
extra work, but for tests with differences per branch it would
mean dealing with conflicts.  In the testsuite I think I'd rather
deal with per branch conditionals over merge with conflicts.
Given the low commit volume, even lower commit volume against existing
tests, high compartmentalization and small nature of the files in the
testsuite,  I don't see conflicts being a big thing.  If there are
any, then they're probably justified.
I just want to propose an alternative way of (mostly)
accomplishing the goal.  We could create a tag '12' from current
master.  This would represent the last revision of the testsuite
known to work with Asterisk 12 and below.  We would then be free
to remove compatibility with EOL Asterisk branches from testsuite
master.  We would tag testsuite '14' soon after September 26th,
2018 (EOL for Asterisk 14). This would avoid multiplying the
number of gerrit reviews for testsuite changes, but it would
require continuing to maintain version declarations for supported
versions of Asterisk.  I think this would be a good trade-off so
we aren't stuck with all the 1.8/11 baggage.
We were thinking of taking the current master, copying it to 13, 14,
15, and "legacy" then starting with the next release our release
scripts would automatically create tags in the testsuite just as they
do in asterisk.
This also makes Jenkins much happier BTW.

Post by Kevin Harwell
Greetings,
We're thinking about adding a branching system to the Asterisk
Testsuite. Each branch would be named the same as, and correspond
to, an Asterisk branch. So for instance the following branches
13, 14, 15
For each release of Asterisk we will also create a tag in the
Testsuite that corresponds to that release's tag. That way
someone could checkout both tags for easy testing
Other advantages? Most all, if not all, the current versioning
stuff found in the Testsuite could go away, or be safely ignored
moving forward. The versioning has become a bit cumbersome
especially when you have to make a backward incompatible change
to a test. Moving the version control out of the Testsuite and
into a version control system should alleviate the need for this
moving forward.
Please let us know your thoughts and considerations on moving
forward with this model. Especially any potential pitfalls or
problems you might see with it.
Thanks!
--
Kevin Harwell
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at:http://digium.com &http://asterisk.org

--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
asterisk-dev mailing list
http://lists.digium.com/mailman/listinfo/asterisk-dev
<http://lists.digium.com/mailman/listinfo/asterisk-dev>
--
George Joseph
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - US
Check us out at: www.digium.com <http://www.digium.com/> &
www.asterisk.org <http://www.asterisk.org/>

[George Joseph]

Sounds good. So after this is done am I correct that when my review
against asterisk 13 is approved jenkins will 'gate' using the 13 branch of
testsuite?

Correct.

Will the testsuite also get minor branches (like 13.19)?

Not quite sure yet. I'd say "no" until there's a good reason to create the
minor branches. As it is, nothing would go into an asterisk minor branch
without going through the mainline branch first. So for 13.19 for
instance, we'd just gate against 13. Now when 13.19.0 is released, we'd
tag the testsuite 13 branch with "13.19.0" then when and if we need to
release 13.19.1, we'd test it with the "13.19.0" tag from the testsuite.

I think. :)

Post by Corey Farrell
It would be nice to strip out some/most of the per version conditionals
in tests. The need to always cherry-pick changes is the only pitfall I see
with your proposal. If a test never had per version differences then the
cherry-pick is trivial amount of extra work, but for tests with differences
per branch it would mean dealing with conflicts. In the testsuite I think
I'd rather deal with per branch conditionals over merge with conflicts.

Given the low commit volume, even lower commit volume against existing
tests, high compartmentalization and small nature of the files in the
testsuite, I don't see conflicts being a big thing. If there are any,
then they're probably justified.

Post by Corey Farrell
I just want to propose an alternative way of (mostly) accomplishing the
goal. We could create a tag '12' from current master. This would
represent the last revision of the testsuite known to work with Asterisk 12
and below. We would then be free to remove compatibility with EOL Asterisk
branches from testsuite master. We would tag testsuite '14' soon after
September 26th, 2018 (EOL for Asterisk 14). This would avoid multiplying
the number of gerrit reviews for testsuite changes, but it would require
continuing to maintain version declarations for supported versions of
Asterisk. I think this would be a good trade-off so we aren't stuck with
all the 1.8/11 baggage.

We were thinking of taking the current master, copying it to 13, 14, 15,
and "legacy" then starting with the next release our release scripts would
automatically create tags in the testsuite just as they do in asterisk.
This also makes Jenkins much happier BTW.

Post by Corey Farrell
Greetings,
We're thinking about adding a branching system to the Asterisk Testsuite.
Each branch would be named the same as, and correspond to, an Asterisk
13, 14, 15
For each release of Asterisk we will also create a tag in the Testsuite
that corresponds to that release's tag. That way someone could checkout
both tags for easy testing
Other advantages? Most all, if not all, the current versioning stuff
found in the Testsuite could go away, or be safely ignored moving forward.
The versioning has become a bit cumbersome especially when you have to make
a backward incompatible change to a test. Moving the version control out of
the Testsuite and into a version control system should alleviate the need
for this moving forward.
Please let us know your thoughts and considerations on moving forward
with this model. Especially any potential pitfalls or problems you might
see with it.
Thanks!
--
Kevin Harwell
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at: http://digium.com & http://asterisk.org
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
asterisk-dev mailing list
http://lists.digium.com/mailman/listinfo/asterisk-dev

--
George Joseph
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - US
Check us out at: www.digium.com & www.asterisk.org
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
asterisk-dev mailing list
http://lists.digium.com/mailman/listinfo/asterisk-dev

[Kevin Harwell]

I went ahead and created an issue to track this:

https://issues.asterisk.org/jira/browse/ASTERISK-27492

Feel free to add comments or suggestions on the issue as well.

Post by Kevin Harwell
Greetings,
We're thinking about adding a branching system to the Asterisk Testsuite.
Each branch would be named the same as, and correspond to, an Asterisk
13, 14, 15
For each release of Asterisk we will also create a tag in the Testsuite
that corresponds to that release's tag. That way someone could checkout
both tags for easy testing
Other advantages? Most all, if not all, the current versioning stuff found
in the Testsuite could go away, or be safely ignored moving forward. The
versioning has become a bit cumbersome especially when you have to make a
backward incompatible change to a test. Moving the version control out of
the Testsuite and into a version control system should alleviate the need
for this moving forward.
Please let us know your thoughts and considerations on moving forward with
this model. Especially any potential pitfalls or problems you might see
with it.
Thanks!
--
Kevin Harwell
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at: http://digium.com & http://asterisk.org

[Kevin Harwell]

I'm getting ready to start the process of branching the testsuite. I'm
going to give a basic overview here of what I'm going to do, so as things
get posted on gerrit people will have some idea as to what is coming, and
why.

To start I'm not going to touch the current master branch. This will be
renamed later to "legacy" once the new branches have all been created and
readied.

The first branch I'll create is the 13 branch. Once created I'm going to
remove all tests that have their min version's marked as < 13.19.0 (current
release of Asterisk). I'll then push this up for review. Once it's accepted
I'll move onto the next step.

Next I will create the 14 branch from the 13 branch. I'll then remove all
tests from the 14 branch that have their min version's marked as < 14.7.5.
I'll also remove all tests from the 13 branch that have a max version > 13.
I'll then push both of these for review.

Once accepted I'll then do similar for the 15 branch. It will be created
from 14, and all tests will be removed that have a min version < 15.2.0.
All tests with a max version > 14 will be removed from the 14 branch. Once
these have been reviewed and accepted I will do the same for what will be
the new master branch (probably temporarily call it that, "new_master",
until we rename the old one to "legacy").

Once all this part is done, or mostly done for a branch, I'll go through
all the tests for each branch and remove the versioning stuff from the yaml
files.

At this point the branches should be in their final ready states. I'll
update the Jenkins scripts to point to the new branches (13, 14, 15). If
everything seems good to go I'll rename the current master to "legacy", and
rename "new_master" to "master". I'll also I'll create a tag for each
branch based on the latest Asterisk version that's been released. At some
point in this process I'll also update the mkrelease script to auto-create
a new tag in the testsuite based on the version of Asterisk being released.

I'm hoping by doing it this way it will minimize the size of the each
review and help separate out the different types of changes.

Thanks!

[Kevin Harwell]

Post by Kevin Harwell
I'm getting ready to start the process of branching the testsuite. I'm
going to give a basic overview here of what I'm going to do, so as things
get posted on gerrit people will have some idea as to what is coming, and
why.
To start I'm not going to touch the current master branch. This will be
renamed later to "legacy" once the new branches have all been created and
readied.
The first branch I'll create is the 13 branch. Once created I'm going to
remove all tests that have their min version's marked as < 13.19.0 (current
release of Asterisk). I'll then push this up for review. Once it's accepted
I'll move onto the next step.
Next I will create the 14 branch from the 13 branch. I'll then remove all
tests from the 14 branch that have their min version's marked as < 14.7.5.
I'll also remove all tests from the 13 branch that have a max version > 13.
I'll then push both of these for review.
Once accepted I'll then do similar for the 15 branch. It will be created
from 14, and all tests will be removed that have a min version < 15.2.0.
All tests with a max version > 14 will be removed from the 14 branch. Once
these have been reviewed and accepted I will do the same for what will be
the new master branch (probably temporarily call it that, "new_master",
until we rename the old one to "legacy").
Once all this part is done, or mostly done for a branch, I'll go through
all the tests for each branch and remove the versioning stuff from the yaml
files.
At this point the branches should be in their final ready states. I'll
update the Jenkins scripts to point to the new branches (13, 14, 15). If
everything seems good to go I'll rename the current master to "legacy", and
rename "new_master" to "master". I'll also I'll create a tag for each
branch based on the latest Asterisk version that's been released. At some
point in this process I'll also update the mkrelease script to auto-create
a new tag in the testsuite based on the version of Asterisk being released.
I'm hoping by doing it this way it will minimize the size of the each
review and help separate out the different types of changes.
Thanks!

Actually, as the head of the testsuite branch is always moving toward the
next release I'm going to remove tests for each branch less than or equal
to the current released version of Asterisk for a branch. That'll make the
new branches starting point align with the following Asterisk versions:

13.20.0
14.7.6
15.3.0

To run any test against Asterisk prior to these versions one will need to
run them against the "legacy" branch.



Kevin Harwell
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at: http://digium.com & http://asterisk.org

[Kevin Harwell]

Update!

Branching now exists in the Asterisk testsuite. Moving forward, when
creating a test for the testsuite you should no longer need to include the
Asterisk versioning information. Just write the test against the
appropriate branch. Also, when uploading a patch to gerrit, please be sure
to cherry pick the test to any other applicable branches.

As well when executing the testsuite be sure to run it against the correct
Asterisk branch. Otherwise you will receive failures due to some tests not
being supported on that branch.

For each new release of Asterisk a tag is created in the Testsuite for the
previous release's branch, which is then deleted (the branch that is).
Note, in order to minimize the number of tags, especially the number of
tags potentially pointing to the same commit, only one final tag is created
for a single release branch. This tag will have the same version number as
the last Asterisk version released for that branch.

Lastly, as mentioned in another post a "legacy" branch now exists that
maintains the state of the testsuite just prior to adding in branching.
This exists for historical purposes and new tests should *not* go into it.

Thanks!

Post by Kevin Harwell
Greetings,
We're thinking about adding a branching system to the Asterisk Testsuite.
Each branch would be named the same as, and correspond to, an Asterisk
13, 14, 15
For each release of Asterisk we will also create a tag in the Testsuite
that corresponds to that release's tag. That way someone could checkout
both tags for easy testing
Other advantages? Most all, if not all, the current versioning stuff found
in the Testsuite could go away, or be safely ignored moving forward. The
versioning has become a bit cumbersome especially when you have to make a
backward incompatible change to a test. Moving the version control out of
the Testsuite and into a version control system should alleviate the need
for this moving forward.
Please let us know your thoughts and considerations on moving forward with
this model. Especially any potential pitfalls or problems you might see
with it.
Thanks!
--
Kevin Harwell
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at: http://digium.com & http://asterisk.org