logstash - Catching a comma seperated pattern with Grok -


i parsing though set of logs 1 field giving me issues. format

header(ip, date etc.) field1=data, field2=data, field3=data, field4=data have general parser read 

match => [ "message","%{data:..header..} %{data}=%{data:service},%{data}=%{data:roles}],%{data}=%{data:macaddress},%{data}=%{data:nasip}"]

some times "value" portion "roles" field looks value, [admin]. handled ] in %{data}=%{data:roles}], in other cases get

subvalue1, subvalue2, subvalue3,

or 

subvalue1, subvalue2, subvalue3, subvalue4,

or

subvalue1, subvalue2,

and parser captures subval1. can see.. there variable number of sub vals , hard catch when ] missing.

here example of kind of log creating issues:

local1--debug--10.47.130.2--2017-03-24--2017-03-24t11:29:51-‌​04:00--11:29:51,545 10.241.186.253 ztp0 session 20 1 0 common.username=labf5chk,common.service=f5_healthchk,common.‌​roles=employee, [user authenticated],common.nas-ip-address=xxxxxxxxxxxx,common.req‌​uest-timestamp=2017-‌​03-24 11:27:56-04

is there work around this?

for variable length comma separated data suggest capturing whole set of values 1 field , parsing field using csv filter.

for parsing set of key=value pairs suggest using kv filter.

so config work this

filter {   grok {     match => [ "message","%{data:..header..} %{greedydata:kv_pairs}"]   }   kv {     source => "kv_pairs"     field_split => ","   }   csv {     # assumes key 'roles'     source => "roles"     target => "role_list"   } }

i not sure of exact format of log messages, kv filter might screw if messages have format, doesn't separate subvalue csv list list of k=v pairs this:

...,key=value,roles=subval1,subval2,subval3,key2=value2...

or opens list [ doesn't close it.

edit: looks though first breaking case in fact you're facing.

if roles section in same place, followed same key, match using

...common.‌​roles=%{data:roles},common.nas-ip-address=%{data:nasip}...

if these kv pairs consitently in same arrangement, using pattern should work. if field @ consistent or matchable more specific regex .*? should use that, use actual key names/patterns instead of %{data}= tempts mismatching.


Comments

Post a Comment

Popular posts from this blog

How to understand 2 main() functions after using uftrace to profile the C++ program? -

i trying uftrace profile following simple c++ program: #include <iostream> class { public: a() {std::cout << "a created" << std::endl;} ~a() {std::cout << "a destroyed" << std::endl;} }; int main() { a; return 0; } the profile result this: # uftrace a.out created destroyed # duration tid function 2.026 [ 4828] | __cxa_atexit(); [ 4828] | main() { [ 4828] | __static_initialization_and_destruction_0() { 89.397 [ 4828] | std::ios_base::init::init(); 0.768 [ 4828] | __cxa_atexit(); 93.029 [ 4828] | } /* __static_initialization_and_destruction_0 */ 94.425 [ 4828] | } /* main */ [ 4828] | main() { [ 4828] | a::a() { 11.104 [ 4828] | std::operator <<(); 10.825 [ 4828] | std::basic_ostream::operator <<(); 24.514 [ 4828] | } /* a::a */ [ 4828] | a::~a() { 0.978 [ 4828] | ...
Read more

c# - Update a combobox from a presenter (MVP) -

i using mvp in project, new in mvp. have 2 comboboxes. when select option in combobox, combobox should filled new data. this action in presenter. view 'view1' in presenter, , introduced combobox1 , combobox2 properties in 'view1', because need 'datasource', 'displaymember', 'valuemember' , 'refresh()' in method below. but, when using pattern, enough send property like public string combobox2 { { return combobox1.selectedvalue.tosstring(); } } into presenter not whole combobox. how can solve problem? public void onselectedindexchangedcombobox1() { if (view1.combobox1.selectedindex == -1) { return; } datatable dt = tools.getdatatable("a path"); var query = (from o in dt.asenumerable() o.field<string>("afield") == farmerview.combobox1.selectedvalue.tostring() orderby o.field<string>("anotherfield") select...
Read more

How to put a lock and transaction on table using spring 4 or above using jdbcTemplate and annotations like @Transactional? -

i trying put lock on table while writing on table , if happened in between roll-back . trying convert below code lock table test_g1 read; lock table test_g write; -- begin; start transaction; insert test_g1 values(143); insert test_g values(145); select * test_g1; select * test_g; rollback; select * test_g; unlock tables; how convert above code @transactional spring jdbctemplate code? @transactional(rollbackfor={dataaccessexception.class}) public void test(){ jdbctemplate.execute("insert test1 (id, nam) values (4, 'a')"); throw new dataaccessexception("error") { }; } here trying throw error, insert statement should rollback it's not happening . thanks edit-1 attaching code , doing in jdbcdaoimpl.java , mentioned problem comment above test() . app.java package com.cgiri.javabrains.spring4; import org.springframework.beans.factory.annotation.autowired; import org.springframework.context.applic...
Read more