如何使用 Kibana Console DSL 查詢語言 - 操作 Elasticsearc 資料結構(上)

如何使用 Kibana Console DSL 查詢語言 - 操作 Elasticsearc 資料結構(上)

在 Kibana Console 可以用來直接執行查詢檢索及操作數據功能,再輸入 Elasticsearch 查詢語言 DSL

在這裡會列出一些實用的 DSL CRUD 查詢語句:

(在這裡的 index 都以 poc-trylog 來做範例說明)

在開始前先複習一下 Elasticsearch 與關聯資料庫概念的類比:

DBMS Elasticsearch
database index
table type (7.0 後預設為 _doc)
row document
column field
schema mapping
SQL DSL (Descriptor structure language)

(在 7.0 之前,同一個 index 可以創建多個類型,7.0 以後,一個索引只能創建一種類型 - _doc)

在這裡我們會以傳輸方法及命令進行說明:

後面的部分我們會在提到 Elasticsearch Analyzer 分詞器是什麼?實際範例演練

GET「 _cat 」查詢支援命令及索引

透過 _cat CVT2HUGO: 命令後,可以透過 ```Ctrl/Cmd CVT2HUGO: 可以用來查詢支援的命令

CVT2HUGO: + Enter``` 送出結果,接下來,分別說明一些常見的查詢方式。
GET _cat

也可列出目前的索引

GET _cat/indeces

輸出結果

green  open apm-7.9.1-span-000004                6lTXIJjISs6MYutfkYHrSQ 1 0  121742      0  27.9mb  27.9mb
green  open apm-7.9.1-metric-000002              JDKWgF3wS1qbqQcKPF-RxA 1 0       0      0    208b    208b
green  open apm-7.9.1-metric-000003              VMKH7L8VRaaGaUA7t2MHnA 1 0   60738      0  12.7mb  12.7mb

GET「 _cat/indexName/?v 」查看詳細內容

透過 ?v 可以用來查看該索引的詳細內容,例如,查看. users_logs 的詳細內容:

GET _cat/users_logs?v

GET [ indexName/_search ] 搜尋索引內容

透過 [indexName]/_search 可以搜尋索引內容

例如

GET poc-trylog/_search

輸出結果如下:

{
  "took" : 1, //搜尋所需要的時間
  "timed_out" : false,
  "_shards" : { //分片,在集群時使用
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,//命中,目前命中(預設為 10000,也就是最多查詢10000個結果)
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "poc-trylog",//索引名稱
        "_type" : "_doc",//索引類型
        "_id" : "PzpcmnYB0g5WnlnpsGRo", //索引ID
        "_score" : 1.0,//權重分數
        "_source" : { //數據本身
          "@version" : "1",
          "@timestamp" : "2020-12-25T14:45:33.542Z",
          "host" : "172.26.0.1",
          "user" : "adam",
          "headers" : {
            "http_accept" : "*/*",
            "content_length" : "96",
            "http_host" : "127.0.0.1:8080",
            "content_type" : "application/json",
            "request_path" : "/logstash-poc-test/1",
            "http_user_agent" : "curl/7.64.1",
            "http_version" : "HTTP/1.1",
            "request_method" : "PUT"
          },
          "post_date" : "2020-12-07T10:12:12",
          "message" : "trying out Elasticsearch"
        }
      },...
    ]
  }
}

GET [ indexName/_count ] 查詢 index 結果總數

透過 _count

CVT2HUGO: 可查詢 index 結果總數,例如:
GET poc-trylog/_count

得出結果

{
  "count" : 3,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  }
}

GET [ indexName/indices ] 查看所有索引

GET [ indexName/_doc/[id] ] 依照 ID 查詢數據

例如,透過以下方式查詢指定ID內容

GET poc-trylog/_doc/PzpcmnYB0g5WnlnpsGRo

輸出結果

{
  "_index" : "poc-trylog",
  "_type" : "_doc",
  "_id" : "PzpcmnYB0g5WnlnpsGRo",
  "_version" : 1,
  "_seq_no" : 0,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "@version" : "1",
    "@timestamp" : "2020-12-25T14:45:33.542Z",
    "host" : "172.26.0.1",
    "user" : "adam",
    "headers" : {
      "http_accept" : "*/*",
      "content_length" : "96",
      "http_host" : "127.0.0.1:8080",
      "content_type" : "application/json",
      "request_path" : "/logstash-poc-test/1",
      "http_user_agent" : "curl/7.64.1",
      "http_version" : "HTTP/1.1",
      "request_method" : "PUT"
    },
    "post_date" : "2020-12-07T10:12:12",
    "message" : "trying out Elasticsearch"
  }
}

POST [ indexName/_doc/[id] ] 增加一筆數據,若存在就覆蓋

可以透過 POST 方式來新增文檔內容,並且可指定ID,如果ID沒有指定就會由Elasticsearch 自動生成。

如果是已經存在的ID,則會覆蓋掉原有ID內容。

POST poc-trylog/_doc/1234
{ "user" : "brown", 
  "post_date" : "2020-12-26T10:12:12",
  "message" : "hello brown" 
}

輸出結果

{
  "_index" : "poc-trylog",
  "_type" : "_doc",
  "_id" : "1234",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 3,
  "_primary_term" : 1
}

POST [ indexName/_create/[id] ] 增加一筆數據,若存在就拒絕

前面我們提到,透過 _doc 新增數據,若ID已經存在就會進行覆蓋。

這裡的 _create 也是新增內容,但id如果存在就會報錯。

POST poc-trylog/_create/1234
{ "user" : "brown", 
  "post_date" : "2020-12-26T10:12:12",
  "message" : "hello brown3" 
}

輸出內容

{
  "error" : {
    "root_cause" : [
      {
        "type" : "version_conflict_engine_exception",
        "reason" : "[1234]: version conflict, document already exists (current version [2])",
        "index_uuid" : "nCu2xyJfSlycXx4e3DzWxw",
        "shard" : "0",
        "index" : "poc-trylog"
      }
    ],
    "type" : "version_conflict_engine_exception",
    "reason" : "[1234]: version conflict, document already exists (current version [2])",
    "index_uuid" : "nCu2xyJfSlycXx4e3DzWxw",
    "shard" : "0",
    "index" : "poc-trylog"
  },
  "status" : 409
}

POST [ indexName/_update/[id] ] 更新數據

更新數據可用 _update 更新內容:

POST poc-trylog/_update/1234
{ "doc": {
  "message" : "hello brown6"
  }
}

輸出結果

{
  "_index" : "poc-trylog",
  "_type" : "_doc",
  "_id" : "1234",
  "_version" : 4,
  "result" : "updated",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 6,
  "_primary_term" : 1
}

PUT [ indexName/_doc/[id] ] 更新數據

與前面提到的 _update

CVT2HUGO: 一樣,可以針對指定ID更新內容
PUT poc-trylog/_doc/1234
{
  "doc": {
    "message" : "hello bb7"
  }
}

輸出結果

{
  "_index" : "poc-trylog",
  "_type" : "_doc",
  "_id" : "1234",
  "_version" : 5,
  "result" : "updated",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 7,
  "_primary_term" : 1
}

PUT [ indexName/_create/[id] ] 增加一筆數據,若存在就拒絕

與前面 __create

如果 ID 存在則拒絕新增:

CVT2HUGO: 一樣,可以指定ID新增數據,若沒有指定則由 Elasticsearch 自動生成,
PUT poc-trylog/_create/1234
{ "user" : "cayla", 
  "post_date" : "2020-12-26T10:12:13",
  "message" : "hello3344" 
}

輸出結果

{
  "error" : {
    "root_cause" : [
      {
        "type" : "version_conflict_engine_exception",
        "reason" : "[1234]: version conflict, document already exists (current version [5])",
        "index_uuid" : "nCu2xyJfSlycXx4e3DzWxw",
        "shard" : "0",
        "index" : "poc-trylog"
      }
    ],
    "type" : "version_conflict_engine_exception",
    "reason" : "[1234]: version conflict, document already exists (current version [5])",
    "index_uuid" : "nCu2xyJfSlycXx4e3DzWxw",
    "shard" : "0",
    "index" : "poc-trylog"
  },
  "status" : 409
}

DELETE [ indexName ] 刪除索引

透過 DELETE 方法可以刪除指定的 索引

DELETE poc-trylog

DELETE [ _doc/[id] ] 刪除數據

透過 DELETE 方法可以刪除指定的 id

DELETE poc-trylog/_doc/11

輸出結果:

{
  "_index" : "poc-trylog",
  "_type" : "_doc",
  "_id" : "11",
  "_version" : 2,
  "result" : "deleted",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 16,
  "_primary_term" : 1
}


POST [ indexName/_bulk ] 批量新增

透過 _bulk

CVT2HUGO: 批量新增方式
POST poc-trylog/_bulk
{"index":{"_id": 11}}
{"user":"a","post_date" : "2020-12-26T10:12:11","message" : "txt1" }
{"index":{"_id": 12}}
{"user":"b","post_date" : "2020-12-26T10:12:12","message" : "txt2" }
{"index":{"_id": 13}}
{"user":"c","post_date" : "2020-12-26T10:12:13","message" : "txt3" }
{"index":{"_id": 14}}
{"user":"d","post_date" : "2020-12-26T10:12:14","message" : "txt4" }

輸出結果

{
  "took" : 6,
  "errors" : false,
  "items" : [
    {
      "index" : {
        "_index" : "poc-trylog",
        "_type" : "_doc",
        "_id" : "11",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 12,
        "_primary_term" : 1,
        "status" : 201
      }
    },
    {
      "index" : {
        "_index" : "poc-trylog",
        "_type" : "_doc",
        "_id" : "12",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 13,
        "_primary_term" : 1,
        "status" : 201
      }
    },
    {
      "index" : {
        "_index" : "poc-trylog",
        "_type" : "_doc",
        "_id" : "13",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 14,
        "_primary_term" : 1,
        "status" : 201
      }
    },
    {
      "index" : {
        "_index" : "poc-trylog",
        "_type" : "_doc",
        "_id" : "14",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 15,
        "_primary_term" : 1,
        "status" : 201
      }
    }
  ]
}

GET [ mget ] 批量+跨索引查詢

mget 批量查詢,可以從不同索引取得數據:

GET _mget
{
	"docs": [
    {"_index": "poc-logs", "_id": 12},
    {"_index": "movies", "_id": 1},
    {"_index": "users", "_id": 2}
  ]
}

備註:Kibana console 快捷鍵 參考

  • Ctrl/Cmd + I

    針對目前查詢自動建立 indent

    Auto indent current request

  • Ctrl/Cmd + /

    開啟目前查詢相關的文件

    Open documentation for current request

  • Ctrl + Space

    開啟自動完成

    Open Auto complete (even if not typing)

  • Ctrl/Cmd + Enter

    送出查詢

    Submit request

  • Ctrl/Cmd + Up/Down

    跳到前一個或下一個查詢的開始或結束

    Jump to the previous/next request start or end.

  • Ctrl/Cmd + Alt + L

    搜集或擴大目前的規模

    Collapse/expand current scope.

  • Ctrl/Cmd + Option + 0

    Collapse all scopes but the current one. Expand by adding a shift.

  • Down arrow

    Switch focus to auto-complete menu. Use arrows to further select a term

  • Enter/Tab

    Select the currently selected or the top most term in auto-complete menu

  • Esc

    關閉 auto-complete 選單

    Close auto-complete menu