Number Crunching the National Finals

[amc]

Now that the Open qualifying round runs over 4 full days, it opens the possibility of adopting a daily qualifying scheme; the top 10 dogs from each day would make it to Saturday. I've become a proponent of daily qualifying because conditions can change dramatically over the course of 4 days. It also affords the opportunity to re-run sheep should that become necessary.

 

So, I took a look at this year's numbers:

 

Tuesday: 37 dogs ran, 14 (37.8%) DQ/RT, average score 142.8

Wednesday: 38 dogs ran, 15 (39.5%) DQ/RT, average score 148.7

Thursday: 36 dogs ran, 14 (38.9%) DQ/RT, average score 145.1

Friday: 38 dogs ran, 15 (39.5%) DQ/RT, average score 131

 

Totals: 149 runs, 58 (38.9%) DQ/RT, average score 141.9

 

I found it very interesting how equal the DQ/RT numbers were. I'll leave it someone else to do the regression analysis, but I'll bet that the difference in Friday's average score is statistically significant.

 

 

If daily qualifying was in effect, 5 dogs from Friday would have made the cut; one from Tuesday, three from Wednesday, and one from Thursday would not have made it in.

 

Amy

[juliepoudrier]

Funny you should post this as I was thinking about daily qualifying the other day. For this finals it seems the weather was pretty consistent across all four days (hot!), but I think that by choosing X dogs from each day, one can take into account changing weather conditions and how that could affect the running (e.g., if it had been cool weather three days and then blistering the fourth day, scores might have been lower on the hotter day, meaning less dogs from that day qualified).

 

Anyway, I *think* today was the hottest day so far all week, and I suppose that could account for lower scores....

 

J.

[MagRam]

Now that the Open qualifying round runs over 4 full days, it opens the possibility of adopting a daily qualifying scheme; the top 10 dogs from each day would make it to Saturday. I've become a proponent of daily qualifying because conditions can change dramatically over the course of 4 days. It also affords the opportunity to re-run sheep should that become necessary.

 

So, I took a look at this year's numbers:

 

Tuesday: 37 dogs ran, 14 (37.8%) DQ/RT, average score 142.8

Wednesday: 38 dogs ran, 15 (39.5%) DQ/RT, average score 148.7

Thursday: 36 dogs ran, 14 (38.9%) DQ/RT, average score 145.1

Friday: 38 dogs ran, 15 (39.5%) DQ/RT, average score 131

 

Totals: 149 runs, 58 (38.9%) DQ/RT, average score 141.9

 

I found it very interesting how equal the DQ/RT numbers were. I'll leave it someone else to do the regression analysis, but I'll bet that the difference in Friday's average score is statistically significant.

If daily qualifying was in effect, 5 dogs from Friday would have made the cut; one from Tuesday, three from Wednesday, and one from Thursday would not have made it in.

 

Amy

 

Interesting analysis and daily qualifying may be a viable alternative (Although I doubt it is adopted soon).

 

However, I think one four day trial is not enough to perform any valid statistical analysis of the hypothesis. It might be interesting to see from multiple years or multiple major trials whether there is a statistical difference but that is obviously a lot of work.

 

Interesting that after about 20 dogs the statistical projected cut line for the top 40 was a score of 153 and it remained that way through the first three days with the actual cut eventually coming at 151. This would suggest taht the distribution of scores (as opposed to the simple arithmetic mean) stayed fairly consistent over 4 days.

[Pearse]

So, I took a look at this year's numbers:

 

Tuesday: 37 dogs ran, 14 (37.8%) DQ/RT, average score 142.8

Wednesday: 38 dogs ran, 15 (39.5%) DQ/RT, average score 148.7

Thursday: 36 dogs ran, 14 (38.9%) DQ/RT, average score 145.1

Friday: 38 dogs ran, 15 (39.5%) DQ/RT, average score 131

 

Totals: 149 runs, 58 (38.9%) DQ/RT, average score 141.9

 

I found it very interesting how equal the DQ/RT numbers were. I'll leave it someone else to do the regression analysis, but I'll bet that the difference in Friday's average score is statistically significant.

 

 

OK, I crunched the numbers using just the scored runs and the differences weren't statistically significant for any combination of days. The closest was Wed compared to Friday where the p-value (unpaired t-test, two-tailed p-value) was just >0.05 so almost significant but not quite. Tues and Thurs were both p > 0.1 when compared to Friday.

 

Then I realized that throwing out the no scores was wrong and introduced bias so I reran the numbers comparing Tues, Wed, and Thurs to Friday using results from all of the runs (adding the RT/DQ zero scores drops the mean almost by half).

 

Tues mean score = 88 standard deviation 73.34 p-value (comparing to Friday) 0.6886

 

Wed mean score = 87.69 Std Dev = 77.09, p-value 0,7074

 

Thu mean score = 91.77 Std Dev = 74.97, p-value 0.5358.

 

Generally, a p-value of less than 0.05 is accepted as showing a statistically significant difference between two groups with 95% confidence. These p-values are 10X higher so there are likely no real significant differences between the scores.

 

(Caveat: I'm not a statistician, I don't even play one on TV, and I did not sleep at a Holiday Inn Express last night).

 

Pearse

[amc]

Thanks, Pearse, I knew you'd jump in with the big guns.

 

So, no statistical significance in the daily numbers, but how could you analyze the fact that 5 handlers from Friday would have made the semis if daily qualifying was in place? Would it make any sense to try to mathematically analyze it? It is just a marginally interesting factoid?

 

Amy

[MagRam]

Wow !!! You guys are taking me back to my college statistical analysis class and that is WAY TOO LONG AGO for me to admit.

 

Ignoring sample size for the moment - and why this would have to be run over multiple trials for it to be significant statistically.

 

However, I think you have a defintional error in your explanation. Assuming you were using daily qualifying that would mean the top 10 from each day would qualify so there would not be only 5 dogs from Friday. By definition and structure of the daily qualifying rule there would be 10 from each day. The relevant question would be how much difference there would be in what score would be required to place top 10 on a particular day. Unfortunately the posted results are now sorted according to score and no longer include the run order number so I do not have complete numbers in my downloads which would show who ran on which day. However, as near as I can tell from the numbers I did download - which did not include about the last seven runs Friday - the daily cut offs for top 10 would have been as follows:

 

Tues - 153

Wed - 144 - 1 dog that did not make it under the current system would have made it in

Thurs - 152

Friday - Unfortunately I did not download the data timely enough to still have run orders available.

 

Obviously (although I guess nothing is truly obvious with statistics) the difference between Tues and Thurs of 1 point is negligible. The Wed group had 9 dogs at 153 or above so this does not seem significant. The gap to the 10th dog was 9 points - thus the 144 score would have been tenth place for the day.

 

The final cut off was 151 and it had projected to that 153 from fairly early Tuesday so it did not change much although it clearly dropped more on Friday than any other day. Overall though it seems the current method worked fairly well although a daily qualifying rule would be interesting and might be exciting (I seem to recall they did this in one of the UK Nationals a couple years ago - or maybe it is that they do it regularly - on the series that was shown on RFD and it was exciting to watch)

 

From following the tweets and the score postings what seemed to the unscientific observation to make more difference than which day was the time of day. Don't know how you would prove or disprove that or what you could do about it anyway.

 

I guess - like so many things in sheepdogging - it comes down to luck of the draw.

[juliepoudrier]

Okay, I think I misunderstand daily qualifying. I thought it meant that a set number from each day went forward to the semis, so in this case, over four days of running 10 dogs from each day would move forward. If that's not the case, and apparently it's not since Amy is saying that just five dogs from Friday would have moved to the next round, tell me how it actually works?

 

J.

[MagRam]

Okay, I think I misunderstand daily qualifying. I thought it meant that a set number from each day went forward to the semis, so in this case, over four days of running 10 dogs from each day would move forward. If that's not the case, and apparently it's not since Amy is saying that just five dogs from Friday would have moved to the next round, tell me how it actually works?

 

J.

 

Julie - I think you have it right. See long post immediately above.

 

I recall from the series they showed on RFD that they did daily qualifying in one of the UK countries (I seem to recall Scotland - may have been England) and it was largely because a hoof and mouth disease impacted their ability to get enough sheep for everyone to have fresh each day.

[MagRam]

Just by comparison I happen to have downloaded the 2010 Meeker scores and sorted to a Spreadsheet by Score and the Daily Results are still up on the Meeker site.

 

Anyway - top 30 overall went through to the semifinals - the cut off (last dog in) was 44. There were 3 days of qualifying run so daily qualifying would mean 10 from each day. Applying Daily qualifying the daily cut offs would have been:

 

Day 1 - 50 (5 dogs that qualified on overall scores running day 1 would not have qualified)

Day 2 - 41 (1 happy team that did make it on overall would have made it in)

Day 3 - 34 (4 happy dog/handler teams that would have made the semis based on daily qualifying) (Plus looking at who some of these 4 are it does not seem to be just random that poor dog/handler teams were running this day as some who would have made it in are pretty darn good teams but were out under the current system)

 

Frankly the difference from day to day seems - without running the ANOVA - to have potentially made a greater difference at Meeker than at the finals.

 

I suppose somebody with a lot of time on their hands could do an analysis of whether their is a random distribution over many multi-day trials but it is frankly well beyond my capacity.

 

Very interesting though.

[Journey]

Unfortunately the posted results are now sorted according to score and no longer include the run order number so I do not have complete numbers in my downloads which would show who ran on which day.

 

I have them if you want it. I am pretty sure I have Dwight Parker in the order that he ran on my sheet too.

[Pearse]

Thanks, Pearse, I knew you'd jump in with the big guns.

 

So, no statistical significance in the daily numbers, but how could you analyze the fact that 5 handlers from Friday would have made the semis if daily qualifying was in place? Would it make any sense to try to mathematically analyze it? It is just a marginally interesting factoid?

 

Amy

 

Well you could argue day to day variation, but the numbers don't support that to be significant.

 

What if you broke the day down into three equal time periods? Would you find that there were statistically significant differences between morning runs, midday runs, and afternoon runs? If so, you could make a stronger argument to take the top 1/3 morning runs, the top 1/3 midday runs, and the top 1/3 afternoon runs. My guess is that you would have a stronger case.

 

Fundamentally, you have too many variables. Sheep can be different hour to hour. Weather can be a factor day to day or just part of one day. You have the luck of the draw. You could be in with the ten top handlers running their better dogs on Tues. Someone on Friday is in with not so many top handlers or those same handlers running their second dogs.

 

To me, the only good argument for top x per day is if there's a need to use rerun sheep, and I think I'd be pretty upset if I got a good score on fresh sheep on Tues and someone else got in with a lower score on rerun dog broke sheep on Friday just because the luck of the draw put me in with better handlers and their best dogs.

 

Otherwise, I think the way it is done now is the fairest method. Best scores go forward.

 

Pearse

[amc]

A couple of points: @Julie...what I meant with the 5 dogs from Friday remark is...if daily qualifying was in effect, there were 5 dogs that ran on Friday that WOULD have made it in based on their Friday only score...but did NOT make it in based on the overall score.

 

@Pearse, WRT having a great run on fresh Tuesday sheep...that should get you 'in' with daily qualifying...you don't have to worry about someone else's score on dog-broke sheep later in the week. Soldier Hollow is the obvious example.

 

Amy

[MagRam]

A couple of points: @Julie...what I meant with the 5 dogs from Friday remark is...if daily qualifying was in effect, there were 5 dogs that ran on Friday that WOULD have made it in based on their Friday only score...but did NOT make it in based on the overall score.

 

@Pearse, WRT having a great run on fresh Tuesday sheep...that should get you 'in' with daily qualifying...you don't have to worry about someone else's score on dog-broke sheep later in the week. Soldier Hollow is the obvious example.

 

Amy

 

Well since you mentioned Soldier Hollow - I also had those downloaded in a spreadsheet - 15 dogs back to the finals - top 5 from each day - so a daily qualifier model.

 

Cut off from -

 

Day 1 - 74 (Faansie and Jill)

Day 2 - 77 (Suzy and Buzz)

Day 3 - 81 (Lavon and Tess)

 

So the rising scores fit the anticipated pattern when rerunning sheep.

 

If done on multiple day scoring - not top 5 each day - just top 15 overall the cutoff would have been 77. Four of the five qualifiers from Day 1 would NOT have qualified. Not sure if this is statistically significant although it seems it would be but I am sure it was significant to those 4 who qualified from Day 1.

 

Interestingly the overall and reserve champion both qualified on Day 1. The overall would barely have qualified on a non-daily qualifier and the reserve champion would NOT have qualified at all.

 

Very interesting discussion.

[Mark Billadeau]

Friday was brutal.

 

Take a look at who made the cut Friday and I'll bet all if not most also made it into the finals.

[amc]

Top 17 by day qualified:

 

Tuesday - 6

Wednesday - 5

Thursday - 3

Friday - 3

 

One team each from Tuesday, Wednesday and Thursday would not have qualified if daily qualifying were in effect.

 

Amy

[juliepoudrier]

^^Is that correct? Wouldn't an extra handler (or two) have qualified on Thursday (and Friday) if day qualifications were in effect (vs. one fewer handler on Thursday, when there were only three to start)?

 

J.

[amc]

^^Is that correct? Wouldn't an extra handler (or two) have qualified on Thursday (and Friday) if day qualifications were in effect (vs. one fewer handler on Thursday, when there were only three to start)?

 

J.

 

Yes, extra handlers would have qualified for the semifinals if daily qualifying were in effect, but the ones I noted that would not have made are actual teams that wouldn't have made it based on daily qualifying. It's analogous to MagRam's comment about Soldier Hollow...the champ would have barely squeaked in and the reserve champ would not have made it in if non-daily qualifying was used.

 

Amy

[Laurae]

But perhaps would different strategies have been used if daily qualifiers were in place? I mean, obviously everyone does as well as they can all the time in a general competitive sense, but I wonder if some handlers (especially given the ongoing heat) did what was necessary to just make it in during their qualifying runs and so may have done things a little differently if a different qualifying scheme was used?

[amc]

But perhaps would different strategies have been used if daily qualifiers were in place? I mean, obviously everyone does as well as they can all the time in a general competitive sense, but I wonder if some handlers (especially given the ongoing heat) did what was necessary to just make it in during their qualifying runs and so may have done things a little differently if a different qualifying scheme was used?

 

That's an interesting question...just do what you think will be the minimum (or just above) to make it to Saturday.

 

Well....with the current system, you need to be in the top 26.6% of dogs to make the top 40 (assuming 150 to the post; this year it was top 26.3% because 149 ran).

 

If you needed to make the top 10 on a 38 dog day, that is 26.3%. Top 10 on a 37 dog day is 27.0%, and top 10 on a 36 dog day would be 27.7%.

 

So, you still need to be in the top quarter regardless, but daily qualifying would smooth out any inequities due to weather, etc. Plus the other benefits, like knowing your fate at the day's end instead of the week's end.

 

Amy

[MagRam]

Plus the other benefits, like knowing your fate at the day's end instead of the week's end.

 

Amy

 

Just to play Devils Advocate - the current method leaves you still with hope if you are 1 or 2 dogs out of making the daily cut. Although if you are making travel plans you may want to get out of Dodge earlyy if you know you are out to keep accomodation costs down.

[Sue R]

Plus the other benefits, like knowing your fate at the day's end instead of the week's end.

 

Amy

Makes me think that it would reduce the suspense (both for competitor and for spectator) and be easier on the competitors. I guess both approaches have their proponents but this is an idea that might deserve consideration for multiple reasons.

[juliepoudrier]

That's an interesting question...just do what you think will be the minimum (or just above) to make it to Saturday.

It's funny that this was brought up because I've often wondered if Alasdair and others at his level didn't do just this--having watched them at big trials. It seems to me from my own observations that the top handlers do well in the preliminary rounds so that they get through, but they save that extra oomph (for lack of a better term) for the final round. For example, maybe they demand precision on the lines, but not as much precision as they would in the final go, as a means of not overtiring their dogs mentally. I have no idea of any of them actually employ such a strategy, and I guess it could backfire, but it would make sense to do so--to build up to the peak performance in the last round.

 

J.

[Bruce Smart]

I have always thought that the reason for going to daily qualifying was that it allowed more flexibility in managing the trial. I recall the World Trial in Ireland. They ran with different flocks on different fields with different judges simultaneously and got the job done in 2 days.

 

It also allows gives trial managers more options in adjusting to natural disasters (weather, sick judge, etc) because the course can be changed from day to day.

 

Bruce

[Laurae]

And if daily qualifiers had been in place last week, they could have put the water tub closer to the pen on Wednesday for the rest of the week.