[ixpmanager] Permission denied from sflow-to-rrd-handler / rrdcached

[Ian Chilton]

Hi,

I’m replacing our aged sflow host with a new one and am having an odd 
problem.

Debian 10 box.

Both rrdcached and sflow-to-rrd-handler are running as root (which I 
would
like to change going forward, but first things first!).

rrdcached is is running from the stock scripts in Debian, other than I
Tweaked /etc/default/rrdcached and set:
BASE_OPTIONS="-B -P FLUSH,UPDATE -m 0666"

/data is a separate XFS mount, mounted with ‘noatime,nodiratime’. 
Have copied
across the data from the old box (120GB of history!) as a starting 
point.

Have added to the sflowtool fanouts on the old box to feed the new box, 
for
testing.

A short time after starting sflow-to-rrd-handler, syslog is filled with:

Feb 18 10:44:04 sflow2 sflow-to-rrd-handler[2113]: WARNING: while 
updating 
/data/ixpmatrix/ipv4/bytes/p2p/src-00175/p2p.ipv4.bytes.src-00175.dst-00334.rrd: 
rrdcached at unix:/var/run/rrdcached.sock: 
/data/ixpmatrix/ipv4/bytes/p2p/src-00175/p2p.ipv4.bytes.src-00175.dst-00334.rrd: 
Permission denied
Feb 18 10:44:04 sflow2 sflow-to-rrd-handler[2113]: WARNING: while 
updating 
/data/ixpmatrix/ipv4/bytes/p2p/src-00175/p2p.ipv4.bytes.src-00175.dst-00436.rrd: 
rrdcached at unix:/var/run/rrdcached.sock: 
/data/ixpmatrix/ipv4/bytes/p2p/src-00175/p2p.ipv4.bytes.src-00175.dst-00436.rrd: 
Permission denied
Feb 18 10:44:04 sflow2 sflow-to-rrd-handler[2113]: WARNING: while 
updating 
/data/ixpmatrix/ipv4/bytes/p2p/src-00175/p2p.ipv4.bytes.src-00175.dst-00389.rrd: 
rrdcached at unix:/var/run/rrdcached.sock: 
/data/ixpmatrix/ipv4/bytes/p2p/src-00175/p2p.ipv4.bytes.src-00175.dst-00389.rrd: 
Permission denied


…which is very odd when both rrdcached and sflow-to-rrd-handler are 
running as root?!

I read the error as rrdcached complaining about writing to the disk.

rrdcached definitely is running as root:
root       450     1  6 10:10 ?        00:00:16 /usr/bin/rrdcached -B -P 
FLUSH,UPDATE -m 0666 -b /var/lib/rrdcached/db/ -j 
/var/lib/rrdcached/journal/ -p /var/run/rrdcached.pid -l 
unix:/var/run/rrdcached.sock

Those various paths all exist:
/var/lib/rrdcached/db/ (is empty?)
/var/lib/rrdcached/journal/
/var/run/rrdcached.pid
/var/run/rrdcached.sock

The sock file is s+666.

I can touch the data files, as well as create/delete files in the dir - 
so not a filesystem problem.

root at sflow2:~# ls -la 
/data/ixpmatrix/ipv4/bytes/p2p/src-00175/p2p.ipv4.bytes.src-00175.dst-00334.rrd
-rw-r--r-- 1 root root 185312 Feb 18 11:33 
/data/ixpmatrix/ipv4/bytes/p2p/src-00175/p2p.ipv4.bytes.src-00175.dst-00334.rrd

If I unmount /data - them it writes the (some) files, but still gives 
the errors in syslog, but after a few seconds it and the logs fill the 
not very big root partition.

If I stop rrdcached and start it manually, with:
rrdcached -P FLUSH,UPDATE -m 0666 -l unix:/var/run/rrdcached.sock

Then it works!

Searching the archives for this list, gives this post from 2013:
https://www.inex.ie/pipermail/ixpmanager/2013-November/000260.html
… but other than the FLUSHALL on -P, is what I’m using.

Anyone seen similar?

Thanks,

Ian