Thursday, March 16, 2017

Import data into InfluxDB



The scenario is:
  1. Some process sends metrics to Telegraf 
  2. For some reason, Telegraf outputs metrics to a file
  3. Import the file from 2) into InfluxDB

 

Telegraf output to a file

Telegraf has a file output plugin, just enable it:
# # Send telegraf metrics to file(s)
 [[outputs.file]]
#   ## Files to write to, "stdout" is a specially handled file.
   files = ["/tmp/metrics.out"]
#
#   ## Data format to output.
   data_format = "influx"

 

Import data into InfluxDB

InfluxDB CLI has a -import argument, the most detailed documentation about how to use it is https://github.com/influxdata/influxdb/tree/master/importer, but it talks about InfluxDB 0.8.9, so that is confusing.  By reading the source code, I figured out the data file has to follow this format (the green lines are my explaination):

# DDL
#if you need to create a database to store your data, put the DDL commands below, e.g.
CREATE DATABASE db0
CREATE DATABASE db1
CREATE RETENTION POLICY rp1 ON db1 DURATION 1h REPLICATION 1

# DML
#specify DB, retention policy for your data below, e.g.
# CONTEXT-DATABASE:db0
# CONTEXT-RETENTION-POLICY:rp1
#and put your data in one-line protacal here, e.g.

mem,host=QAVM107 active=0i,available=15573110784i,available_percent=90.64989902698733 1489567540000000000


I already had the database created, so I just need to insert these lines into the file generated by Telegraf:

# DML
# CONTEXT-DATABASE:test

Now run –import:
influx -import -path=/tmp/metrics.out -precision=s
2017/03/16 15:56:47 error writing batch:  {"error":"unable to parse 
 'mem,host=QAVM107 active=0i,available=15573110784i,available_percent=90.64989902698733 1489567540000000000': 
time outside range -9223372036854775806 - 9223372036854775806"}
 
This is most strange, to prove that the data point is in the correct format, I inserted it directly:
influx -database telegraf -execute 'insert mem,host=QAVM107 active=0i,available=15573110784i,available_percent=90.64989902698733 1489567540000000000'

Sure this is successful, proving that the data point string is in the correct format and the timestamp is not outside of the time range. 

To find out what is wrong, I enabled all logs in /etc/influxdb/influxdb.conf, and restarted influxdb:
sudo service influxd stop
influxd

Nothing interesting got logged. 

As a last resort, I tried this command without –precision argument and it worked!
influx -import -path=/tmp/metrics.out 
 
So that is the whole story. Without going into more details in the source code, I can’t tell why –precision argument caused this issue.
 






 

1 comment:

  1. I can tell you why. In the -prcision argument you specified seconds with "-precision=s" standard is nanoseconds or ns and your data also is in ns so specifying that wrongly made it fail.

    ReplyDelete