Elasticsearch 常見索引操作彙整

這裡記錄一些 Elasticsearch 常用的索引操作方式，方便之後查詢參考

創建 shopping 索引

在創建 index 如果沒有額外設定，預設會是一個 shard 及一個 replica

request PUT 
'http://localhost:9200/shopping'

response

{
    "acknowledged": true,
    "shards_acknowledged": true,
    "index": "shopping"
}

創建 products 索引同時設定 shards 與 replicas

在創建索引時，可以同時設置 shards 以及 replicas，例如下方為 3 main shards 以及 replicas shards

request PUT 'http://localhost:9200/products' 
header 'Content-Type: application/json' 
data-raw '{
  "settings":{
    "number_of_shards": 3,
    "number_of_replicas": 1
  }
}'

response

{
    "acknowledged": true,
    "shards_acknowledged": true,
    "index": "products"
}

設定完成後，如果是單一節點，上面 3 個 shards 都會分配到同一個 node 中

可以透過 chrome 瀏覽器的擴充套件 elasticsearch-head 來查看 elasticsearch cluster 的狀況。

查詢索引及設定

request GET 
'http://localhost:9200/products'

查看目前所有索引

request GET 
'http://localhost:9200/_cat/indices?v'

response

health status index    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   shopping exwbCoeHTv28rHU2cASxhg   1   1          0            0       208b           208b

刪除 shopping 索引

request DELETE 
'http://localhost:9200/shopping'

response

{
    "acknowledged": true
}

新增 document 到 shopping index

request POST 
'http://localhost:9200/shopping/_doc' 
header 'Content-Type: application/json'
data-raw '{
    "title":"青年教育基金",
    "category":"基金",
    "links":"https://resource.example",
    "amounts":12000.00
}'

response

{
    "_index": "shopping",
    "_type": "_doc",
    "_id": "IZWY8n0B-m-l2jCaD2Gt",
    "_version": 1,
    "result": "created",
    "_shards": {
        "total": 2,
        "successful": 1,
        "failed": 0
    },
    "_seq_no": 0,
    "_primary_term": 1
}

新增一筆 document 並且指定 index

request POST 
'http://localhost:9200/shopping/_doc/0001'
header 'Content-Type: application/json'
data-raw '{
    "title":"青年教育基金",
    "category":"基金",
    "links":"https://resource.example",
    "amounts":12000.00
}'

查詢 shopping document 並且指定 index

curl --location --request GET 'http://localhost:9200/shopping/_doc/0001'

rsponse

{
    "_index": "shopping",
    "_type": "_doc",
    "_id": "0001",
    "_version": 6,
    "_seq_no": 10,
    "_primary_term": 1,
    "found": true,
    "_source": {
        "title": "青年教育基金",
        "category": "基金",
        "links": "https://resource.example",
        "amounts": 12000.00
    }
}

查詢 shopping index 底下所有數據

request GET 
'http://localhost:9200/shopping/_search'

resonse

{
    "took": 14,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 7,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [
            {
                "_index": "shopping",
                "_type": "_doc",
                "_id": "IZWY8n0B-m-l2jCaD2Gt",
                "_score": 1.0,
                "_source": {
                    "title": "青年教育基金",
                    "category": "基金",
                    "links": "https://resource.example",
                    "amounts": 12000.00
                }
            },
            {
                "_index": "shopping",
                "_type": "_doc",
                "_id": "IpWb8n0B-m-l2jCap2Ex",
                "_score": 1.0,
                "_source": {
                    "title": "青年教育基金",
                    "category": "基金",
                    "links": "https://resource.example",
                    "amounts": 12000.00
                }
            },
            {
                "_index": "shopping",
                "_type": "_doc",
                "_id": "I5Wc8n0B-m-l2jCaCGEK",
                "_score": 1.0,
                "_source": {
                    "title": "青年教育基金",
                    "category": "基金",
                    "links": "https://resource.example",
                    "amounts": 12000.00
                }
            },
            {
                "_index": "shopping",
                "_type": "_doc",
                "_id": "JJWc8n0B-m-l2jCaC2FF",
                "_score": 1.0,
                "_source": {
                    "title": "青年教育基金",
                    "category": "基金",
                    "links": "https://resource.example",
                    "amounts": 12000.00
                }
            },
            {
                "_index": "shopping",
                "_type": "_doc",
                "_id": "JZWc8n0B-m-l2jCaDWFw",
                "_score": 1.0,
                "_source": {
                    "title": "青年教育基金",
                    "category": "基金",
                    "links": "https://resource.example",
                    "amounts": 12000.00
                }
            },
            {
                "_index": "shopping",
                "_type": "_doc",
                "_id": "0001",
                "_score": 1.0,
                "_source": {
                    "title": "青年教育基金",
                    "category": "基金",
                    "links": "https://resource.example",
                    "amounts": 12000.00
                }
            },
            {
                "_index": "shopping",
                "_type": "_doc",
                "_id": "0002",
                "_score": 1.0,
                "_source": {
                    "title": "青年教育基金",
                    "category": "基金",
                    "links": "https://resource.example",
                    "amounts": 12000.00
                }
            }
        ]
    }
}

全量資料更新，shopping 指定 index 完全更新單筆資料

request PUT 
'http://localhost:9200/shopping/_doc/0001'
header 'Content-Type: application/json'
data-raw '{
    "title":"青年教育基金",
    "category":"基金",
    "links":"https://resource.example",
    "amounts":12222.00
}'

response

{
    "_index": "shopping",
    "_type": "_doc",
    "_id": "0001",
    "_version": 7,
    "result": "updated",
    "_shards": {
        "total": 2,
        "successful": 1,
        "failed": 0
    },
    "_seq_no": 17,
    "_primary_term": 1
}

更新指定 index 的指定 fields

request POST 'http://localhost:9200/shopping/_update/0001' 
header 'Content-Type: application/json' 
data-raw '{
    "doc": {
        "title":"中年創業基金"
    }
}'

response

{
    "_index": "shopping",
    "_type": "_doc",
    "_id": "0001",
    "_version": 8,
    "result": "updated",
    "_shards": {
        "total": 2,
        "successful": 1,
        "failed": 0
    },
    "_seq_no": 18,
    "_primary_term": 1
}

document 刪除指定 index

request DELETE 'http://localhost:9200/shopping/_doc/0001'

response

{
    "_index": "shopping",
    "_type": "_doc",
    "_id": "0001",
    "_version": 9,
    "result": "deleted",
    "_shards": {
        "total": 2,
        "successful": 1,
        "failed": 0
    },
    "_seq_no": 19,
    "_primary_term": 1
}

查詢指定 fields

request GET 
'http://localhost:9200/shopping/_search?q=title:中'

response

{
    "took": 4,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 1.4816045,
        "hits": [
            {
                "_index": "shopping",
                "_type": "_doc",
                "_id": "0001",
                "_score": 1.4816045,
                "_source": {
                    "title": "中年創業基金",
                    "category": "基金",
                    "links": "https://resource.example",
                    "amounts": 12222.0
                }
            }
        ]
    }
}

透過 body 來進行指定 field 查詢（推薦）

預設的查詢可以使用 match 來查詢欄位，在 Elasticsearch 會將查詢的內容進行分詞處理，再進行索引查詢，所以底下查詢 中青會被分詞處理成 中, 青 , 中青 來查詢

request GET 
'http://localhost:9200/shopping/_search'
header 'Content-Type: application/json'
data-raw '{
    "query": {
        "match":{
            "title":"中青"
        }
    }
}'

response

{
    "took": 4,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 7,
            "relation": "eq"
        },
        "max_score": 1.3862942,
        "hits": [
            {
                "_index": "shopping",
                "_type": "_doc",
                "_id": "0001",
                "_score": 1.3862942,
                "_source": {
                    "title": "中年創業基金",
                    "category": "基金",
                    "links": "https://resource.example",
                    "amounts": 12222.0
                }
            },
            {
                "_index": "shopping",
                "_type": "_doc",
                "_id": "IZWY8n0B-m-l2jCaD2Gt",
                "_score": 0.2876821,
                "_source": {
                    "title": "青年教育基金",
                    "category": "基金",
                    "links": "https://resource.example",
                    "amounts": 12000.00
                }
            },
            ....
        ]
    }
}

分頁

可以透過 from 開始位置以及 size 來決定要顯示的筆數

request GET 
'http://localhost:9200/shopping/_search' 
header 'Content-Type: application/json' 
data-raw '{
    "query": {
        "match":{
            "title":"青"
        }
    },
    "from": 2,
    "size": 2
}'

指定顯示欄位

可以透過 _source 來指定顯示欄位

request GET 
'http://localhost:9200/shopping/_search' 
header 'Content-Type: application/json' 
data-raw '{
    "query": {
        "match":{
            "title":"青"
        }
    },
    "from": 2,
    "size": 2,
    "_source":["title"]
}'

指定排序方式

可以透過 sort 來指定欄位，以及 order 規則

request GET 
'http://localhost:9200/shopping/_search'
header 'Content-Type: application/json'
data-raw '{
    "query": {
        "match":{
            "title":"青"
        }
    },
    "from": 2,
    "size": 2,
    "_source":["title"],
    "sort": {
        "amounts": {
            "order": "desc"
        }
    }
}'

多條件查詢

這裡一樣使用 GET http://localhost:9200/shopping/_search ，並且在 BODY 傳送查詢條件內文

AND

{
    "query":{
        "bool":{ //表示多個條件
            "must":[ //多個條件必須同時成立 (AND)
                {
                    "match":{ //匹配條件
                        "title":"中年"
                    }
                },
                {
                    "match":{
                        "amounts":12222.00
                    }
                }
            ]
        }
    }
}

{
    "query":{
        "bool":{ //表示多個條件
            "should":[ //多個條件共同查詢 (OR)
                {
                    "match":{ //匹配條件
                        "title":"中年"
                    }
                },
                {
                    "match":{
                        "title":"青年"
                    }
                }
            ]
        }
    }
}

RANGE

{
    "query":{
        "bool":{ //表示多個條件
            "should":[ //多個條件共同查詢 (OR)
                {
                    "match":{ //匹配條件
                        "title":"中年"
                    }
                },
                {
                    "match":{
                        "title":"青年"
                    }
                }
            ],
            "filter":{//過濾器
                "range":{//查詢範圍條件
                    "amounts":{
                        "gt": 12200//大於
                    }
                }
            }
        }
    }
}

完全匹配查詢

Elasticsearch 預設會將查詢字串經過分詞再查詢，如果要根據輸入內文查詢，可以透過 `query.match_phrase"，會忽略分詞處理的流程

{
    "query": {
        "match_phrase":{
            "title":"中"
        }
    }
}

高亮顯示

可以使用 highlight 針對分詞查詢的位置進行標記

{
    "query": {
        "match_phrase":{
            "title":"中"
        }
    },
    "highlight":{//高亮顯示
        "fields":{
            "title":{} //依照title欄位來顯示
        }
    }
}

response

{
    "took": 96,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 1.3862942,
        "hits": [
            {
                "_index": "shopping",
                "_type": "_doc",
                "_id": "0001",
                "_score": 1.3862942,
                "_source": {
                    "title": "中年創業基金",
                    "category": "基金",
                    "links": "https://resource.example",
                    "amounts": 12222.0
                },
                "highlight": {
                    "title": [
                        "<em>中</em>年創業基金"
                    ]
                }
            }
        ]
    }
}

聚合操作查詢

可以透過 aggs 對查詢結果進行統計分析或分組，可以設定分組名稱以及計算方式(terms 計算數量, avg 計算平均)，以及要查詢的欄位。

由於聚合操作預設還是會返回匹配到的 document，可以透過 size 設定為 0 ，不顯示原始數據。

{
    "aggs":{//聚合操作
            "amounts_group":{//分組名稱
                "terms":{//分組方式：計算數量
                    "field":"amounts" //分組字段
                }
            },"amounts_avg":{//分組名稱
                "avg":{//分組方式：計算平均
                    "field":"amounts" //分組字段
                }
            }
    },
    "size":0 //顯示 document 數據量，設定0表示不顯示原始數據

}

Mapping

在創建索引後，有些欄位需要分詞查詢，有些欄位不需要，可以透過 mapping 來指定 Fields 的規則，主要可以透過 type 來決定是否要進行分詞處理，index 來設定是否儲存到索引庫。

(注意，Mapping 創建後就不能修改)

request: PUT 
path: 'http://localhost:9200/member/_mapping'
header: 'Content-Type: application/json'

data-raw :

{
    "properties":{
        "name":{
            "type":"text",//文本,進行分詞處理
            "index": true //儲存到索引庫，可以被索引查詢
        },
        "sex":{
            "type":"keyword",//不做分詞處理，需要完全匹配
            "index":true //儲存到索引庫，可以被索引查詢
        },
        "phone":{
            "type":"keyword", //不做分詞處理
            "index":false //不建立索引，不可透過索引查詢，但還是會出現在 document 
        }
    }
}

response

{
    "acknowledged": true
}

查詢索引的 Mapping

curl --location --request GET 'http://localhost:9200/member/_mapping'

response

{
    "member": {
        "mappings": {
            "properties": {
                "name": {
                    "type": "text"
                },
                "phone": {
                    "type": "keyword",
                    "index": false
                },
                "sex": {
                    "type": "keyword"
                }
            }
        }
    }
}

Template & Mapping

在 Elasticsearch 有 Dynamic mapping 自動建立索引的功能，也就是 schemaless

但是透過 Dynamic mapping 儲存時，有些型別可能會跟你預設的不一樣

例如，存入數據為 “true”，Dynamic mapping 可能會判斷成 string

這時就會建議透過建立索引時，設定好 Mapping ，也就是定義好 schema 以避免有些型別與預期不一樣。

[Template] 字面上就是模板，也就是可以重複被使用在 Elasticsearch 的 Template 使用時機是，在 Dynamic mapping 時，會根據 index 正則規則是否符合現有模板，如果正則匹配到 template，就會用這個模板來建立索引，當然，template 中也會有 mapping 的資訊。

在 Elasticsearch 官方，預設就有建立以下幾個 template，所以在 log & matrics, synthetics 索引建立時，就會自動透過 common shcema 來

logs-*-*
metrics-*-*
synthetics-*-*

當然在建立 Template 時，可以參考 Elastic Common Schema (ECS) 來將資料做更好的規範。