logstash - how enable anonymous access in ElasticSearch 5.3 -

i have downloaded elasticsearch, logstash , kibana version 5.3 (until few hours ago using 5.2.something). installed xpack in each of elk. after can't use logstash anymore.

logstash error:

./logstash -f /log_to_elastic53.conf

...     [2017-04-06t19:25:55,704][warn ][logstash.outputs.elasticsearch] attempted resurrect connection dead es instance, got error. {:url=>#<uri::http:0x3c6582db url:http://127.0.0.1:9200/>, :error_type=>logstash::outputs::elasticsearch::httpclient::pool::badresponsecodeerror, :error=>"got response code '401' contacting elasticsearch @ url 'http://127.0.0.1:9200/'"} 

my log_to_elastic53.conf

input { stdin { } } output {   elasticsearch { hosts => ["127.0.0.1:9200"] }   stdout { codec => rubydebug } } 

i setup elasticsearch accept anonymous call way:

source idea: https://www.elastic.co/guide/en/x-pack/current/anonymous-access.html

elasticsearch.yml

xpack.security.authc:   anonymous:     username: anonymous_user      roles: role1, role2      authz_exception: false 

p.s. tried both false/true in authz_exception

interesting kibana complaining permission wondering if related debian elk: failed extract phantom.js archive

./kibana

undefined accessed autoload lists no longer available via plugin api.use `ui/autoload/*` modules instead. undefined accessed autoload lists no longer available via plugin api.use `ui/autoload/*` modules instead.   log   [22:24:55.244] [warning] plugin "sense" disabled because expected kibana version "2.0.0-snapshot", , found "5.3.0".   log   [22:24:55.499] [info][status][plugin:kibana@5.3.0] status changed uninitialized green - ready   log   [22:24:55.568] [info][status][plugin:elasticsearch@5.3.0] status changed uninitialized yellow - waiting elasticsearch   log   [22:24:55.575] [info][status][plugin:xpack_main@5.3.0] status changed uninitialized yellow - waiting elasticsearch   log   [22:24:55.739] [info][status][plugin:graph@5.3.0] status changed uninitialized yellow - waiting elasticsearch   log   [22:24:55.747] [info][status][plugin:monitoring@5.3.0] status changed uninitialized green - ready   log   [22:24:55.751] [warning][reporting] generating random key xpack.reporting.encryptionkey. prevent pending reports failing on restart, please set xpack.reporting.encryptionkey in kibana.yml   log   [22:24:55.756] [info][status][plugin:reporting@5.3.0] status changed uninitialized yellow - waiting elasticsearch   log   [22:24:55.958] [error][reporting] extracterror: failed extract phantom.js archive     @ extract.<anonymous> (/home/demetrio/servers/dbs/kibana-5.3.0-amd64/usr/share/kibana/plugins/x-pack/plugins/reporting/server/lib/extract/bunzip2.js:18:16)     @ emitone (events.js:101:20)     @ extract.emit (events.js:188:7)     @ extract.destroy (/home/demetrio/servers/dbs/kibana-5.3.0-amd64/usr/share/kibana/plugins/x-pack/node_modules/tar-stream/extract.js:191:17)     @ onunlock (/home/demetrio/servers/dbs/kibana-5.3.0-amd64/usr/share/kibana/plugins/x-pack/node_modules/tar-stream/extract.js:69:26)     @ stat (/home/demetrio/servers/dbs/kibana-5.3.0-amd64/usr/share/kibana/plugins/x-pack/node_modules/tar-fs/index.js:232:23)     @ /home/demetrio/servers/dbs/kibana-5.3.0-amd64/usr/share/kibana/plugins/x-pack/node_modules/mkdirp/index.js:46:53     @ fsreqwrap.oncomplete (fs.js:123:15)   log   [22:24:55.959] [error][reporting] error: eacces: permission denied, mkdir '/var/lib/kibana/phantomjs-2.1.1-linux-x86_64'     @ error (native)   log   [22:24:55.960] [error][status][plugin:reporting@5.3.0] status changed yellow red - insufficient permissions extracting phantom.js archive. make sure kibana data directory (path.data) owned same user running kibana.   log   [22:24:55.968] [info][status][plugin:security@5.3.0] status changed uninitialized yellow - waiting elasticsearch   log   [22:24:55.969] [warning][security] generating random key xpack.security.encryptionkey. prevent sessions being invalidated on restart, please set xpack.security.encryptionkey in kibana.yml   log   [22:24:55.972] [warning][security] session cookies transmitted on insecure connections. not recommended.   log   [22:24:56.022] [info][status][plugin:searchprofiler@5.3.0] status changed uninitialized yellow - waiting elasticsearch   log   [22:24:56.033] [info][status][plugin:tilemap@5.3.0] status changed uninitialized yellow - waiting elasticsearch   log   [22:24:56.042] [info][status][plugin:console@5.3.0] status changed uninitialized green - ready   log   [22:24:56.217] [info][status][plugin:elasticsearch@5.3.0] status changed yellow green - kibana index ready   log   [22:24:56.219] [info][status][plugin:timelion@5.3.0] status changed uninitialized green - ready   log   [22:24:56.223] [info][listening] server running @ http://localhost:5601   log   [22:24:56.225] [info][status][ui settings] status changed uninitialized green - ready   log   [22:24:56.355] [info][license][xpack] imported license information elasticsearch: mode: trial | status: active | expiry date: 2017-05-06t18:53:19-03:00   log   [22:24:56.365] [info][status][plugin:monitoring@5.3.0] status changed green yellow - waiting monitoring health check   log   [22:24:56.368] [info][status][plugin:xpack_main@5.3.0] status changed yellow green - ready   log   [22:24:56.369] [info][status][plugin:graph@5.3.0] status changed yellow green - ready   log   [22:24:56.370] [info][status][plugin:reporting@5.3.0] status changed red green - ready   log   [22:24:56.371] [info][status][plugin:security@5.3.0] status changed yellow green - ready   log   [22:24:56.371] [info][status][plugin:searchprofiler@5.3.0] status changed yellow green - ready   log   [22:24:56.372] [info][status][plugin:tilemap@5.3.0] status changed yellow green - ready   log   [22:24:58.357] [info][status][plugin:monitoring@5.3.0] status changed yellow green - ready 

instead of allowing anonymous access has high security risk, follow the official documentation configure role , user logstash connect elasticsearch.

logstash needs able manage index templates, create indices, , write , delete documents in indices creates.

to set authentication credentials logstash:

1. create logstash_writer role has manage_index_templates cluster privilege, , write, delete, , create_index privileges logstash indices. can create roles management > roles ui in kibana or through role api:

   post _xpack/security/role/logstash_writer {   "cluster": ["manage_index_templates", "monitor"],   "indices": [     {       "names": [ "logstash-*" ],        "privileges": ["write","delete","create_index"]     }   ] } 

2. create logstash_internal user , assign logstash_writer role. can create users management > users ui in kibana or through user api:

   post _xpack/security/user/logstash_internal {   "password" : "changeme",   "roles" : [ "logstash_writer"],   "full_name" : "internal logstash user" } 

3. configure logstash authenticate logstash_internal user created. configure credentials separately each of elasticsearch plugins in logstash .conf file. example:

   input {     ...     user => logstash_internal     password => changeme   } filter {     ...     user => logstash_internal     password => changeme   } output {   elasticsearch {     ...     user => logstash_internal     password => changeme   }