Problem with BIND 9.10.1-P1 recursion limits

[Stuart Henderson]

The new recursion limits (or at least the default values for them) seem
to have some problems. Simple example, if I start named for recursive
service, no forwarders, debugging enabled, and run "dig @::1 www.ibm.com a"
I get a failure with numerous "exceeded max queries" log entries for gtld
servers etc.

If people are having trouble replicating I can provide more logs off-list
on request (~7K lines for a 'trace 9' log so not included here).

09-Dec-2014 13:44:40.571 exceeded max queries resolving 'g.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.572 exceeded max queries resolving 'm.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.573 exceeded max queries resolving 'm.gtld-servers.net/A'
09-Dec-2014 13:44:40.573 exceeded max queries resolving 'h.gtld-servers.net/A'
09-Dec-2014 13:44:40.574 exceeded max queries resolving 'd.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.575 exceeded max queries resolving 'm.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.668 exceeded max queries resolving 'h.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.668 exceeded max queries resolving 'm.gtld-servers.net/A'
09-Dec-2014 13:44:40.669 exceeded max queries resolving 'm.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.671 exceeded max queries resolving 'd.gtld-servers.net/A'
09-Dec-2014 13:44:40.671 exceeded max queries resolving 'm.gtld-servers.net/A'
09-Dec-2014 13:44:40.672 exceeded max queries resolving 'm.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.690 exceeded max queries resolving 'h.gtld-servers.net/A'
09-Dec-2014 13:44:40.690 exceeded max queries resolving 'm.gtld-servers.net/A'
09-Dec-2014 13:44:40.690 exceeded max queries resolving 'm.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.693 exceeded max queries resolving 'h.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.693 exceeded max queries resolving 'm.gtld-servers.net/A'
09-Dec-2014 13:44:40.695 exceeded max queries resolving 'm.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.699 exceeded max queries resolving 'd.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.699 exceeded max queries resolving 'm.gtld-servers.net/A'
09-Dec-2014 13:44:40.699 exceeded max queries resolving 'i.gtld-servers.net/A'
09-Dec-2014 13:44:40.701 exceeded max queries resolving 'm.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.705 exceeded max queries resolving 'e.gtld-servers.net/A'
09-Dec-2014 13:44:40.705 exceeded max queries resolving 'i.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.705 exceeded max queries resolving 'm.gtld-servers.net/A'
09-Dec-2014 13:44:40.710 exceeded max queries resolving 'e.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.711 exceeded max queries resolving 'j.gtld-servers.net/A'
09-Dec-2014 13:44:40.711 exceeded max queries resolving 'f.gtld-servers.net/A'
09-Dec-2014 13:44:40.712 exceeded max queries resolving 'm.gtld-servers.net/A'
09-Dec-2014 13:44:40.714 exceeded max queries resolving 'm.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.717 exceeded max queries resolving 'j.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.717 exceeded max queries resolving 'g.gtld-servers.net/A'
09-Dec-2014 13:44:40.717 exceeded max queries resolving 'f.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.722 exceeded max queries resolving 'k.gtld-servers.net/A'
09-Dec-2014 13:44:40.722 exceeded max queries resolving 'm.gtld-servers.net/A'
09-Dec-2014 13:44:40.723 exceeded max queries resolving 'g.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.724 exceeded max queries resolving 'm.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.728 exceeded max queries resolving 'k.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.728 exceeded max queries resolving 'l.gtld-servers.net/A'
09-Dec-2014 13:44:40.731 exceeded max queries resolving 'l.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.798 exceeded max queries resolving 'i.gtld-servers.net/A'
09-Dec-2014 13:44:40.803 exceeded max queries resolving 'c.gtld-servers.net/A'
09-Dec-2014 13:44:40.807 exceeded max queries resolving 'i.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.808 exceeded max queries resolving 'd.gtld-servers.net/A'
09-Dec-2014 13:44:40.810 exceeded max queries resolving 'j.gtld-servers.net/A'
09-Dec-2014 13:44:40.812 exceeded max queries resolving 'j.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.815 exceeded max queries resolving 'c.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.817 exceeded max queries resolving 'k.gtld-servers.net/A'
09-Dec-2014 13:44:40.820 exceeded max queries resolving 'k.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.827 exceeded max queries resolving 'e.gtld-servers.net/A'
09-Dec-2014 13:44:40.829 exceeded max queries resolving 'l.gtld-servers.net/A'
09-Dec-2014 13:44:40.833 exceeded max queries resolving 'f.gtld-servers.net/A'
09-Dec-2014 13:44:40.834 exceeded max queries resolving 'g.gtld-servers.net/A'
09-Dec-2014 13:44:40.837 exceeded max queries resolving 'l.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.839 exceeded max queries resolving 'e.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.840 exceeded max queries resolving 'm.gtld-servers.net/A'
09-Dec-2014 13:44:40.843 exceeded max queries resolving 'f.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.996 exceeded max queries resolving 'm.gtld-servers.net/A'
09-Dec-2014 13:44:40.997 exceeded max queries resolving 'eur2.akam.net/A'
09-Dec-2014 13:44:40.997 exceeded max queries resolving 'eur5.akam.net/A'
09-Dec-2014 13:44:40.998 exceeded max queries resolving 'm.gtld-servers.net/AAAA'
09-Dec-2014 13:44:40.998 exceeded max queries resolving 'eur2.akam.net/AAAA'
09-Dec-2014 13:44:40.999 exceeded max queries resolving 'eur5.akam.net/AAAA'
09-Dec-2014 13:44:41.002 exceeded max queries resolving 'usc2.akam.net/A'
09-Dec-2014 13:44:41.002 exceeded max queries resolving 'usc3.akam.net/A'
09-Dec-2014 13:44:41.004 exceeded max queries resolving 'usc2.akam.net/AAAA'
09-Dec-2014 13:44:41.004 exceeded max queries resolving 'usc3.akam.net/AAAA'
09-Dec-2014 13:44:41.005 exceeded max queries resolving 'usw2.akam.net/A'
09-Dec-2014 13:44:41.006 exceeded max queries resolving 'asia3.akam.net/A'
09-Dec-2014 13:44:41.007 exceeded max queries resolving 'usw2.akam.net/AAAA'
09-Dec-2014 13:44:41.008 exceeded max queries resolving 'asia3.akam.net/AAAA'
09-Dec-2014 13:44:41.008 exceeded max queries resolving 'ns1-99.akam.net/A'
09-Dec-2014 13:44:41.008 exceeded max queries resolving 'ns1-206.akam.net/A'
09-Dec-2014 13:44:41.010 exceeded max queries resolving 'ns1-99.akam.net/AAAA'
09-Dec-2014 13:44:41.010 exceeded max queries resolving 'ns1-206.akam.net/AAAA'
09-Dec-2014 13:44:41.011 exceeded max queries resolving 'm.gtld-servers.net/A'
09-Dec-2014 13:44:41.012 exceeded max queries resolving 'm.gtld-servers.net/AAAA'



_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

[Evan Hunt]

Post by Stuart Henderson
The new recursion limits (or at least the default values for them) seem
to have some problems. Simple example, if I start named for recursive
I get a failure with numerous "exceeded max queries" log entries for gtld
servers etc.

Lack of convinction that we'd chosen the correct defaults is the reason
we made them configurable. You can adjust max-recursion-queries if you want
to.

However, in this case I think it's because you had an empty cache, and
sending a second query will clear the problem up. In a future release, we
may want to lift the restrictions temporarily while priming.

[Tony Finch]

Post by Evan Hunt
However, in this case I think it's because you had an empty cache, and
sending a second query will clear the problem up. In a future release, we
may want to lift the restrictions temporarily while priming.

Yes, I could reproduce it after flushing my cache. Had to wait five
minutes before the queries succeeded, which seems unpleasantly long.
I don't know where that time comes from - the ARM says the default
servfail-ttl is 10s.

Tony.
--
f.anthony.n.finch <***@dotat.at> http://dotat.at/
Faeroes: Southwesterly gale 8 to storm 10, occasionally violent storm 11 at
first in east. High becoming very high, occasionally phenomenal later. Rain,
then squally wintry showers. Moderate, occasionally poor.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

[Evan Hunt]

Post by Tony Finch
Yes, I could reproduce it after flushing my cache. Had to wait five
minutes before the queries succeeded, which seems unpleasantly long.
I don't know where that time comes from - the ARM says the default
servfail-ttl is 10s.

You're running unreleased code, there. "Servfail-ttl" is a feature slated
for 9.11, but the recursion limits have only been added in the past few
weeks as a patch for the infinite DNS bug, and we're clearly going to have
to modify the SERVFAIL caching feature in light of this new reality. (We
might arrange for SERVFAILs that occur as a result of recursion limits not
to be cached.)

When I tested this on 9.9, I got the problem with www.ibm.com on the first
query, but it succeeded on the second.

[Stuart Henderson]

Post by Evan Hunt

Post by Tony Finch
Yes, I could reproduce it after flushing my cache. Had to wait five
minutes before the queries succeeded, which seems unpleasantly long.
I don't know where that time comes from - the ARM says the default
servfail-ttl is 10s.

You're running unreleased code, there. "Servfail-ttl" is a feature slated
for 9.11, but the recursion limits have only been added in the past few
weeks as a patch for the infinite DNS bug, and we're clearly going to have
to modify the SERVFAIL caching feature in light of this new reality. (We
might arrange for SERVFAILs that occur as a result of recursion limits not
to be cached.)
When I tested this on 9.9, I got the problem with www.ibm.com on the first
query, but it succeeded on the second.

It's 5 minutes with 9.10.1-P1 as well.

_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

[Evan Hunt]

Post by Stuart Henderson
It's 5 minutes with 9.10.1-P1 as well.

That's unexpected. I'll see if I can reproduce it.

[Evan Hunt]

Post by Evan Hunt
That's unexpected. I'll see if I can reproduce it.

Okay, I can.

Part of the problem is the somewhat crazypants DNS configuration
of www.ibm.com:

$ dig +noall +answer www.ibm.com
www.ibm.com. 3600 IN CNAME www.ibm.com.cs186.net.
www.ibm.com.cs186.net. 60 IN CNAME china-cdn.san.ibm.com.edgekey.net.
china-cdn.san.ibm.com.edgekey.net. 21600 IN CNAME china-cdn.san.ibm.com.edgekey.net.globalredir.akadns.net.
china-cdn.san.ibm.com.edgekey.net.globalredir.akadns.net. 900 IN CNAME e7826.x.akamaiedge.net.
e7826.x.akamaiedge.net. 20 IN A 23.59.201.136

... like, *wow*. A chain of five aliases with TTLs ranging from 20
seconds to 6 hours, passing through five different zones (ibm.com,
cs186.net, edgekey.net, akadns.net, akamaiedge.net), hosted by
servers in three *more* zones (ihost.com, akam.net, and akadns.org,
in addition to akadns.net and akamaiedge.net). I had to almost
double the maximum recursion queries to 99 to get this to work on
an empty cache. Yikes.

Almost any non-empty cache will dodge the bullet. Preceeding the
lookup of www.ibm.com with "dig @::1 ns com" causes the query to
succeed. Also, as previously noted, on 9.9 it will succeed without
a five-minute delay if you just issue the query a second time.

So, possible workarounds if this issue is causing problems for you:

- Ensure that the first query sent to a newly-primed recursive
resolver isn't quite as spectacular as this one;
- Add "max-recursion-queries 100;" to your options statement;
- Run 9.9.6-P1 instead of 9.10.1-P1

The five-minute delay is still a bit of a puzzle. It happens because
of this code in adb.c:

/* XXXMLG Don't pound on bad servers. */
if (address_type == DNS_ADBFIND_INET) {
name->expire_v4 = ISC_MIN(name->expire_v4, now + 300);
name->fetch_err = FIND_ERR_FAILURE;
inc_stats(adb, dns_resstatscounter_gluefetchv4fail);
} else {
name->expire_v6 = ISC_MIN(name->expire_v6, now + 300);
name->fetch6_err = FIND_ERR_FAILURE;
inc_stats(adb, dns_resstatscounter_gluefetchv6fail);
}

The "now + 300" bit is where the five minutes comes from. That's code
that's been around for years, and it is in 9.9, but apparently it's
reached more easily in 9.10. I'm looking into the reasons for this.

The problem should be addressed in 9.10.2, which is likely to be
released next month.

[Mike Hoskins (michoski)]

Thanks for digging in so fast. Our mitigation will be sticking to
9.9.6-P1, since we like ESV anyway.

Wanted to point out that (perhaps sadly) this isn't so crazypants...or at
least not uncommon. The *edge* and *aka* references speak Akamai DNS+CDN.
From my last overview, this has gotten cleaner in the latest versions of
their offerings -- but many of the large(est) sites on the Internet will
be configured this way today.

-----Original Message-----
From: Evan Hunt <***@isc.org>
Date: Tuesday, December 9, 2014 at 2:41 PM
To: Stuart Henderson <***@spacehopper.org>
Cc: Tony Finch <***@dotat.at>, "bind-***@lists.isc.org"
<bind-***@lists.isc.org>
Subject: Re: Problem with BIND 9.10.1-P1 recursion limits

Post by Evan Hunt

Post by Evan Hunt
That's unexpected. I'll see if I can reproduce it.

Okay, I can.
Part of the problem is the somewhat crazypants DNS configuration
$ dig +noall +answer www.ibm.com
www.ibm.com. 3600 IN CNAME www.ibm.com.cs186.net.
www.ibm.com.cs186.net. 60 IN CNAME
china-cdn.san.ibm.com.edgekey.net.
china-cdn.san.ibm.com.edgekey.net. 21600 IN CNAME
china-cdn.san.ibm.com.edgekey.net.globalredir.akadns.net.
china-cdn.san.ibm.com.edgekey.net.globalredir.akadns.net. 900 IN CNAME
e7826.x.akamaiedge.net.
e7826.x.akamaiedge.net. 20 IN A 23.59.201.136
... like, *wow*. A chain of five aliases with TTLs ranging from 20
seconds to 6 hours, passing through five different zones (ibm.com,
cs186.net, edgekey.net, akadns.net, akamaiedge.net), hosted by
servers in three *more* zones (ihost.com, akam.net, and akadns.org,
in addition to akadns.net and akamaiedge.net). I had to almost
double the maximum recursion queries to 99 to get this to work on
an empty cache. Yikes.
Almost any non-empty cache will dodge the bullet. Preceeding the
succeed. Also, as previously noted, on 9.9 it will succeed without
a five-minute delay if you just issue the query a second time.
- Ensure that the first query sent to a newly-primed recursive
resolver isn't quite as spectacular as this one;
- Add "max-recursion-queries 100;" to your options statement;
- Run 9.9.6-P1 instead of 9.10.1-P1
The five-minute delay is still a bit of a puzzle. It happens because
/* XXXMLG Don't pound on bad servers. */
if (address_type == DNS_ADBFIND_INET) {
name->expire_v4 = ISC_MIN(name->expire_v4, now + 300);
name->fetch_err = FIND_ERR_FAILURE;
inc_stats(adb, dns_resstatscounter_gluefetchv4fail);
} else {
name->expire_v6 = ISC_MIN(name->expire_v6, now + 300);
name->fetch6_err = FIND_ERR_FAILURE;
inc_stats(adb, dns_resstatscounter_gluefetchv6fail);
}
The "now + 300" bit is where the five minutes comes from. That's code
that's been around for years, and it is in 9.9, but apparently it's
reached more easily in 9.10. I'm looking into the reasons for this.
The problem should be addressed in 9.10.2, which is likely to be
released next month.
--
Internet Systems Consortium, Inc.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list
bind-users mailing list
https://lists.isc.org/mailman/listinfo/bind-users

_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

[Charles Swiger]

Hi--

Post by Mike Hoskins (michoski)
Wanted to point out that (perhaps sadly) this isn't so crazypants...or at
least not uncommon. The *edge* and *aka* references speak Akamai DNS+CDN.
From my last overview, this has gotten cleaner in the latest versions of
their offerings -- but many of the large(est) sites on the Internet will
be configured this way today.

Agreed, that looks to be a fairly standard Akamai setup to me as well.

If nothing else, a long CNAME chain with short TTLs provides a decent test case. :-)

Regards,

[David A. Evans]

How does the max-recursion-queries counter interact with DNSSEC validation
and RPZ validation? Are the queries for these checks included in the
max-recursion-queries count or are they in a separate queue?


Why I am asking:
I've been running through my test of the new code and getting a few hits
on domains that I think resolve without these limits. I've have more
work to do on validating if these domains didn't resolve because of some
momentary network or external DNS resolution issue or something related to
these new thresholds. (or in the case of 4842b.y.dotnxdomain.net, Cricket
out registering new domains just to mess with us.)

My test is ~5 million unique A record lookups I'm pushing to a test server
at ~300 q/s.
It has a lengthy RPZ enabled and DNSSec validation on.
After reading this thread, I'm flushing the cache every 30 mins.

I'm getting a handful of these messages, some are just broken domains but
a handful of them seem to resolve on DNS servers on older bind code. They
do not seem to be timed with the cache clearing.

Dec 9 13:59:11 198.206.x.x named[13525]: exceeded max queries resolving
'growthcentre.org/A'
Dec 9 14:15:05 198.206.x.x named[13525]: exceeded max queries resolving
'megadeth.rockmetal.art.pl/A'
Dec 9 14:22:33 198.206.x.x named[13525]: exceeded max queries resolving
'ns3.iplay.net/A'
Dec 10 03:18:54 198.206.x.x named[13525]: exceeded max queries resolving
'4842b.y.dotnxdomain.net/DNSKEY'
Dec 10 03:59:02 198.206.x.x named[13525]: exceeded max queries resolving
'dsl-188-34-202-200.asretelecom.net/A'
Dec 10 03:59:03 198.206.x.x named[13525]: exceeded max queries resolving
'ns1.asretelecom.com/A'
Dec 10 08:19:15 198.206.x.x named[13525]: exceeded max queries resolving
'knurow.eu.org/A'
Dec 10 08:27:36 198.206.x.x named[13525]: exceeded max queries resolving
'lb.z.optimix.asia/NS'
Dec 10 08:31:04 198.206.x.x named[13525]: exceeded max queries resolving
'NS4-AUTH.ALLTEL.NET/A'



David A. Evans
Enterprise IP/DNS Management
Network Infrastructure Tools and Services




From: Evan Hunt <***@isc.org>
To: Stuart Henderson <***@spacehopper.org>
Cc: Tony Finch <***@dotat.at>, bind-***@lists.isc.org
Date: 12/09/2014 01:41 PM
Subject: Re: Problem with BIND 9.10.1-P1 recursion limits

Post by Evan Hunt
That's unexpected. I'll see if I can reproduce it.

Okay, I can.

Part of the problem is the somewhat crazypants DNS configuration
of www.ibm.com:

$ dig +noall +answer www.ibm.com
www.ibm.com. 3600 IN CNAME www.ibm.com.cs186.net.
www.ibm.com.cs186.net. 60 IN CNAME
china-cdn.san.ibm.com.edgekey.net.
china-cdn.san.ibm.com.edgekey.net. 21600 IN CNAME
china-cdn.san.ibm.com.edgekey.net.globalredir.akadns.net.
china-cdn.san.ibm.com.edgekey.net.globalredir.akadns.net. 900 IN CNAME
e7826.x.akamaiedge.net.
e7826.x.akamaiedge.net. 20 IN A 23.59.201.136

... like, *wow*. A chain of five aliases with TTLs ranging from 20
seconds to 6 hours, passing through five different zones (ibm.com,
cs186.net, edgekey.net, akadns.net, akamaiedge.net), hosted by
servers in three *more* zones (ihost.com, akam.net, and akadns.org,
in addition to akadns.net and akamaiedge.net). I had to almost
double the maximum recursion queries to 99 to get this to work on
an empty cache. Yikes.

Almost any non-empty cache will dodge the bullet. Preceeding the
lookup of www.ibm.com with "dig @::1 ns com" causes the query to
succeed. Also, as previously noted, on 9.9 it will succeed without
a five-minute delay if you just issue the query a second time.

So, possible workarounds if this issue is causing problems for you:

- Ensure that the first query sent to a newly-primed recursive
resolver isn't quite as spectacular as this one;
- Add "max-recursion-queries 100;" to your options statement;
- Run 9.9.6-P1 instead of 9.10.1-P1

The five-minute delay is still a bit of a puzzle. It happens because
of this code in adb.c:

/* XXXMLG Don't pound on bad servers. */
if (address_type == DNS_ADBFIND_INET) {
name->expire_v4 = ISC_MIN(name->expire_v4, now + 300);
name->fetch_err = FIND_ERR_FAILURE;
inc_stats(adb, dns_resstatscounter_gluefetchv4fail);
} else {
name->expire_v6 = ISC_MIN(name->expire_v6, now + 300);
name->fetch6_err = FIND_ERR_FAILURE;
inc_stats(adb, dns_resstatscounter_gluefetchv6fail);
}

The "now + 300" bit is where the five minutes comes from. That's code
that's been around for years, and it is in 9.9, but apparently it's
reached more easily in 9.10. I'm looking into the reasons for this.

The problem should be addressed in 9.10.2, which is likely to be
released next month.
--
Evan Hunt -- ***@isc.org
Internet Systems Consortium, Inc.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to
unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users