Automatic flushing of the jnl files

[E***@banque-france.fr]

Hello bind-users,

Tried to find the info on my own but couldn’t get accurate understanding of jnl files flushing mechanism.

Bind-9.10.0-02.P2
RHEL 5.7
Master DNS server, Chrooted env, receiving updates via nsupdate.

Maybe I didn’t catch the memo but I thought the standard way bind handled the dynamic journal (.jnl) files was this way:

All changes made to a zone using dynamic update are written to the zone's journal file. The server will periodically flush the complete contents of the updated zone to its zone file this happens approximately every 15 minutes. NOTE this is from zytrax.com dns documentation so I’m pretty sure this is not from bind 9.10

Doesn’t seem to be the case right now and I’m wondering if my server conf is to blame.

I’m able to sync to flat files using rndc:

rndc sync –clean

That I see in my logs:

named[3001]: received control channel command 'sync -clean'
named[3001]: dumping all zones, removing journal files: success

But the next day I’m seeing tons of jnl files again. From updates of course.

I do have these journal messages in my logs:

general: debug 1: zone XX.128.in-addr.arpa/IN: journal rollforward completed successfully: no journal
general: debug 3: no journal file, but that's OK
general: debug 1: zone XX.XX.10.in-addr.arpa/IN: journal rollforward completed successfully: no journal
general: debug 3: no journal file, but that's OK
general: debug 1: zone XX.XX.10.in-addr.arpa/IN: journal rollforward completed successfully: no journal
general: debug 3: no journal file, but that's OK
general: debug 1: zone XX.168.192.in-addr.arpa/IN: journal rollforward completed successfully: no journal

So it it does seem to be rolling the changes but jnl files still persist. It’s not terribly bothering but I would like to know if this is the normal behavior.

I do not have dnssec-enable of managed-keys-directory set in named.conf

Thanks for your help.

E. Berthiaume

**************************************************************
Ce courrier électronique, y compris les piÚces jointes, est à l'attention exclusive des destinataires désignés et revêt un caractÚre confidentiel.
Si vous recevez ce courrier électronique par erreur, merci d'en informer sans délai l'expéditeur et de supprimer son contenu et ses piÚces jointes.

Le contenu de ce courrier électronique ne pourrait engager la responsabilité de la Banque de France que s'il a été émis par une personne dûment habilitée agissant dans le strict cadre des fonctions auxquelles elle est employée et à des fins non étrangÚres à ses attributions.

L'expéditeur de ce courrier électronique ne peut pas garantir la sécurité de l'acheminement par voie électronique et ne saurait dÚs lors encourir une quelconque responsabilité en cas d'altération de son contenu.

**************************************************************

This e-mail, attachments included, is intended solely for the addressees and should be considered as confidential.
Should you receive this message by error, please notify the sender immediately and destroy this e-mail and its attachments.

Banque de France cannot be considered as liable for the content of this message unless the sender has been duly authorized and has acted strictly in the course of his/her tasks as an employee.

The sender of this e-mail cannot ensure the security of its electronic transmission and consequently will not be liable should its content be altered.
**************************************************************

[Phil Mayers]

Post by E***@banque-france.fr
So it it does seem to be rolling the changes but jnl files still
persist. It’s not terribly bothering but I would like to know if this
is the normal behavior.

It's normal. The .jnl files contain the data required to perform
incremental outbound zone transfers.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

[Howard, Christopher]

The journal files get flushed to the zone file periodically, but old
transactions don't get removed so the journal file will continue to grow
forever. If you're like me and on virtual machines with limited hard disk
capacity, you can limit the journal file size with the max-journal-size
configuration statement. Just make sure that the size is large enough to
hold all of the transactions between flushes (I believe that's around
every 15 minutes). Otherwise, after a crash you would be missing records.

-Christopher

Post by Phil Mayers

Post by E***@banque-france.fr
So it it does seem to be rolling the changes but jnl files still
persist. It¹s not terribly bothering but I would like to know if this
is the normal behavior.

It's normal. The .jnl files contain the data required to perform
incremental outbound zone transfers.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list
bind-users mailing list
https://lists.isc.org/mailman/listinfo/bind-users

_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

[Chris Thompson]

Post by Howard, Christopher
The journal files get flushed to the zone file periodically, but old
transactions don't get removed so the journal file will continue to grow
forever. If you're like me and on virtual machines with limited hard disk
capacity, you can limit the journal file size with the max-journal-size
configuration statement. Just make sure that the size is large enough to
hold all of the transactions between flushes (I believe that's around
every 15 minutes). Otherwise, after a crash you would be missing records.

I am fairly sure you are wrong on that last point. BIND will not flush
journal file entries that have not yet been committed to the master file,
even if they make the journal bigger than max-journal-size. If you specify
"max-journal-size 512;" you will find the journal gets emptied completely,
but only after the master file has been updated. (Of course, as Phil Mayers
points out, this would cause downstream IXFRs to become AXFRs,)
--
Chris Thompson
Email: ***@cam.ac.uk

_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

[Howard, Christopher]

Oh, well that's good to know. :)

-Christopher

Post by Chris Thompson

Post by Howard, Christopher
The journal files get flushed to the zone file periodically, but old
transactions don't get removed so the journal file will continue to grow
forever. If you're like me and on virtual machines with limited hard disk
capacity, you can limit the journal file size with the max-journal-size
configuration statement. Just make sure that the size is large enough to
hold all of the transactions between flushes (I believe that's around
every 15 minutes). Otherwise, after a crash you would be missing records.

I am fairly sure you are wrong on that last point. BIND will not flush
journal file entries that have not yet been committed to the master file,
even if they make the journal bigger than max-journal-size. If you specify
"max-journal-size 512;" you will find the journal gets emptied completely,
but only after the master file has been updated. (Of course, as Phil Mayers
points out, this would cause downstream IXFRs to become AXFRs,)
--
Chris Thompson
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list
bind-users mailing list
https://lists.isc.org/mailman/listinfo/bind-users

_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

[Tony Finch]

I got annoyed by having to manually tune max-journal-size so I had a hack
at the problem:

https://git.csx.cam.ac.uk/x/ucs/ipreg/bind9.git/commitdiff/c8f083b797f9810f

Tony.
--
f.anthony.n.finch <***@dotat.at> http://dotat.at/
Viking: Southeast, veering south or southwest later, 4 or 5, increasing 6 at
times. Moderate or rough. Wintry showers. Good, occasionally poor.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

[Bob Harold]

I like that solution.

I assume that "twice the zone file size" is because half of the entries are
deletes? Do deletes get sent in IXFR? Or is it that typically half of the
journal entries are SOA records?

I just took a peek at my journal files and I see one that is 100 times the
zone file size. I wish the entries had dates, even if just as a comment -
it would be a good log of changes, and I would be able to see how far back
in history the journal went.
--
Bob Harold
hostmaster, UMnet, ITcom
Information and Technology Services (ITS)
***@umich.edu
734-647-6524 desk

Post by Tony Finch
I got annoyed by having to manually tune max-journal-size so I had a hack
https://git.csx.cam.ac.uk/x/ucs/ipreg/bind9.git/commitdiff/c8f083b797f9810f
Tony.
--
Viking: Southeast, veering south or southwest later, 4 or 5, increasing 6 at
times. Moderate or rough. Wintry showers. Good, occasionally poor.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to
unsubscribe from this list
bind-users mailing list
https://lists.isc.org/mailman/listinfo/bind-users

[Tony Finch]

Yes, an IXFR is a series of deletes and adds, which both quote whole records. If you re-sign a zone the IXFR can be nearly twice what an AXFR would be! But in fact the factor of two in my patch comes from the journal compaction logic, which halves the size of the journal. So my patch allows the journal to oscillate between the size of the zone and twice that. Smaller journals may mean you have to AXFR when IXFR would be better.

If you use serial-update-method unixtime or date-based serial numbers then you might be able to get the information you want from the journal.

Tony.
--

Post by Bob Harold
I like that solution.
I assume that "twice the zone file size" is because half of the entries are deletes? Do deletes get sent in IXFR? Or is it that typically half of the journal entries are SOA records?
I just took a peek at my journal files and I see one that is 100 times the zone file size. I wish the entries had dates, even if just as a comment - it would be a good log of changes, and I would be able to see how far back in history the journal went.

_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

[Stuart Browne]

Bob,

Some date and record number details from one of my systems, with 'max-journal-size: 100m'. Yes, I've changed the zone names.. ;)

NOTE: Add/Del numbers show total / non-dnssec-or-soa related update numbers.

'zone1' is a monitoring test zone that has sub-zone delegation changes a few times a minute:
dnssec signed: Yes, NSEC3 with Optout
zone1 size: 127k
one1.jnl size: 63M
date now: Wed Jan 21 22:12:15 UTC 2015
oldest jnl soa: Tue Jan 20 20:42:29 UTC 2015
total records: 1,233
no. SOA del's: 52,964
no. del's: 236,556 / 79,716
no. add's: 236,604 / 79,713

'zone2' is a public delegation zone that changes as customers demand:
dnssec signed: Yes, NSEC3 with Optout
zone2 size: 176M
zone2.jnl size: 100M
date now: Wed Jan 21 22:15:15 UTC 2015
oldest jnl soa: Fri Dec 19 17:22:20 UTC 2014
total records: 5,917,482
no. SOA del's: 138,752
no. del's: 456,870 / 172,427
no. add's: 478,940 / 194,541

'zone3' is a public authoritative zone that rarely changes:
dnssec signed: Yes, NSEC3 with Optout
zone3 size: 1.6M
zone3.jnl size: 69M
date now: Wed Jan 21 22:27:53 UTC 2015
oldest jnl soa: Mon Oct 27 21:19:38 UTC 2014
total records: 6,984
no. SOA del's: 35,144
no. del's: 175,832 / 0
no. add's: 175,832 / 0

So the journal files can live for quite a while ;)

Stuart

-----Original Message-----
Sent: Thursday, 22 January 2015 6:34 AM
To: Bob Harold
Subject: Re: Automatic flushing of the jnl files
Yes, an IXFR is a series of deletes and adds, which both quote whole
records. If you re-sign a zone the IXFR can be nearly twice what an AXFR
would be! But in fact the factor of two in my patch comes from the journal
compaction logic, which halves the size of the journal. So my patch allows
the journal to oscillate between the size of the zone and twice that.
Smaller journals may mean you have to AXFR when IXFR would be better.
If you use serial-update-method unixtime or date-based serial numbers then
you might be able to get the information you want from the journal.
Tony.
--

Post by Bob Harold
I like that solution.
I assume that "twice the zone file size" is because half of the entries

are deletes? Do deletes get sent in IXFR? Or is it that typically half
of the journal entries are SOA records?

Post by Bob Harold
I just took a peek at my journal files and I see one that is 100 times

the zone file size. I wish the entries had dates, even if just as a
comment - it would be a good log of changes, and I would be able to see
how far back in history the journal went.
_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to
unsubscribe from this list
bind-users mailing list
https://lists.isc.org/mailman/listinfo/bind-users

_______________________________________________
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list

bind-users mailing list
bind-***@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users

[E***@banque-france.fr]

Thanks all for your replies. Lots of interesting information!

I’ll keep a eye out for those jnl files.

Thanks again!

E. Berthiaume
**************************************************************
Ce courrier électronique, y compris les piÚces jointes, est à l'attention exclusive des destinataires désignés et revêt un caractÚre confidentiel.
Si vous recevez ce courrier électronique par erreur, merci d'en informer sans délai l'expéditeur et de supprimer son contenu et ses piÚces jointes.

Le contenu de ce courrier électronique ne pourrait engager la responsabilité de la Banque de France que s'il a été émis par une personne dûment habilitée agissant dans le strict cadre des fonctions auxquelles elle est employée et à des fins non étrangÚres à ses attributions.

L'expéditeur de ce courrier électronique ne peut pas garantir la sécurité de l'acheminement par voie électronique et ne saurait dÚs lors encourir une quelconque responsabilité en cas d'altération de son contenu.

**************************************************************

This e-mail, attachments included, is intended solely for the addressees and should be considered as confidential.
Should you receive this message by error, please notify the sender immediately and destroy this e-mail and its attachments.

Banque de France cannot be considered as liable for the content of this message unless the sender has been duly authorized and has acted strictly in the course of his/her tasks as an employee.

The sender of this e-mail cannot ensure the security of its electronic transmission and consequently will not be liable should its content be altered.
**************************************************************