# elasticsearch-engine
**Repository Path**: my-source-project/elasticsearch-engine
## Basic Information
- **Project Name**: elasticsearch-engine
- **Description**: elasticsearch-engine是基于 HighLevelRestClient 封装的 ElasticSearch 查询引擎框架. 支持ElasticSearch基于注解的结构化查询; 基于sql语句的方式查询; 并整合常见的ORM框架, 提供基于ORM框架的Mapper接口自动生成ElasticSearch Sql查询语句,并执行ElasticSearch查询;
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No
## Statistics
- **Stars**: 10
- **Forks**: 0
- **Created**: 2022-04-09
- **Last Updated**: 2025-09-13
## Categories & Tags
**Categories**: Uncategorized
**Tags**: ElasticSearch, MyBatis, JPA, Jooq, elasticsearch查询
## README
# elasticsearch-engine
## 介绍

elasticsearch-engine是基于 HighLevelRestClient 封装的 ElasticSearch 查询引擎框架. 支持ElasticSearch基于注解的结构化查询;
基于sql语句的方式查询;
并整合常见的ORM框架, 提供基于ORM框架的Mapper接口自动生成ElasticSearch Sql查询语句,并执行ElasticSearch查询;
**在需要查询 ElasticSearch 的Mapper接口标注一个注解即可实现 ElasticSearch 查询,无需额外的代码开发;
并可以通过配置中心配置动态切换ElasticSearch和Mysql之间的查询,
实现ElasticSearch查询降级.**
github地址: https://github.com/wanghuan9/elasticsearch-engine
## 主要功能特性
1. 基于注解的方式实现elasticsearch的查询
2. 基于sql语句的方式实现elasticsearch的查询
3. 基于mybatis mapper接口 自动生成elasticsearch查询,并支持数据库回表查询
4. 基于jpa repository接口 自动生成elasticsearch查询,并支持数据库回表查询
5. 基于jooq dao实现类 自动生成elasticsearch查询,并支持数据库回表查询
## 架构模块
1. elasticsearch-engine-base 提供注解查询,sql语句查询,ORM查询sql解析,sql改写等基础功能
2. elasticsearch-engine-mybatis 基于mybatis拦截器 实现sql拦截,改写,执行elasticsearch查询
3. elasticsearch-engine-jpa 基于aop,hibernate sql拦截器以及重新jpa参数绑定模块 实现sql拦截,改写,执行elasticsearch查询
4. elasticsearch-engine-jooq 基于aop,jooq执行监听器 实现sql拦截,改写,执行elasticsearch查询
## 使用说明
所有完整示例 请参考 [使用示例](https://gitee.com/my-source-project/elasticsearch-engine-demo)
### 1.注解查询
#### 1.1复杂参数
1)添加maven依赖
```xml
com.elasticsearch.engine
elasticsearch-engine-base
0.0.1-SNAPSHOT
```
2)定义查询model
```java
package com.elasticsearch.engine.demo.dto.query;
import com.elasticsearch.engine.base.mapping.annotation.*;
import com.elasticsearch.engine.base.mapping.model.extend.PageParam;
import com.elasticsearch.engine.base.mapping.model.extend.RangeParam;
import com.elasticsearch.engine.base.mapping.model.extend.SignParam;
import com.elasticsearch.engine.base.model.annotion.Base;
import com.elasticsearch.engine.base.model.annotion.EsQueryIndex;
import com.elasticsearch.engine.base.model.annotion.Ignore;
import com.elasticsearch.engine.base.model.emenu.EsConnector;
import lombok.Data;
import java.math.BigDecimal;
import java.time.LocalDateTime;
import java.util.List;
/**
* @author wanghuan
* @description: 解析查询注解基础测试
* @mail 958721894@qq.com
* @date 2022-05-31 22:40
*/
@EsQueryIndex(value = "person_es_index")
@Data
public class PersonBaseQuery {
@Term
private BigDecimal salary;
@Terms(value = @Base("item_no"))
private List personNos;
@Terms
private List personNoList;
@Range(value = @Base(value = "status", connect = EsConnector.SHOULD), tag = Range.LE_GE)
private RangeParam rangeStatus;
@Range
private RangeParam createTime;
@WildCard
private String address;
@Prefix
private String personName;
@To(@Base("create_time"))
private LocalDateTime createTimeEnd;
@From(value = @Base("create_time"))
private LocalDateTime createTimeStart;
@PageAndOrder
private PageParam pageParam;
/**
* 标记注解不解析value,只解析注解值
* 需要设置 value值不为空,查询条件才会生效, 但是设置的value不会被解析,仅仅标记是否添加该条件
* 所以value可以任意设置, 但是注意 string 不能为空串,数组类型不能为null
*
* SignParam 表示一种无需解析参数值得 类型
* 也可以使用 Sign.DEFAULT_STRING 表示
*/
@Sort
private SignParam sortStatus;
@Aggs(value = @Base("status"), type = Aggs.COUNT_DESC)
private SignParam groupStatus;
/**
* 表示忽略某个字段 ,被忽略的字段 无论属性值是否为空, 查询时都不会被解析
*/
@Ignore
private String token;
}
```
3)声明查询接口
```java
@EsQueryIndex("person_es_index")
public interface PersonEsModelRepository extends BaseESRepository {
/**
* queryByMode
*
* @param param
* @return
*/
List queryByMode(PersonBaseQuery param);
}
```
4)测试示例
```java
@Slf4j
@RunWith(SpringRunner.class)
@SpringBootTest
public class EsEngineProxyModelQueryTest {
@Resource
private PersonEsModelRepository personEsModelRepository;
/**
* model查询测试
*/
@Test
public void queryByModelTest() {
PersonBaseQuery person = new PersonBaseQuery();
person.setPageParam(PageParam.builderPage().currentPage(1).pageSize(100).build());
person.setSalary(new BigDecimal("67700"));
person.setPersonName("张");
person.setAddress("天府");
person.setCreateTimeStart(LocalDateTime.now().minusDays(300));
person.setCreateTimeEnd(LocalDateTime.now());
List res = personEsModelRepository.queryByMode(person);
log.info("res:{}", JsonParser.asJson(res));
}
}
```
5)查询效果
```json
{
"from": 0,
"size": 100,
"timeout": "10s",
"query": {
"bool": {
"filter": [
{
"wildcard": {
"address": {
"wildcard": "*天府*"
}
}
},
{
"prefix": {
"personName": {
"value": "张"
}
}
},
{
"term": {
"salary": {
"value": 67700
}
}
},
{
"range": {
"create_time": {
"from": "2021-08-23T21:17:23.385Z",
"to": "2022-06-19T21:17:23.385Z",
"include_lower": true,
"include_upper": true,
"time_zone": "+08:00",
"format": "8uuuu-MM-dd'T'HH:mm:ss.SSS'Z'"
}
}
}
]
}
}
}
```
#### 1.2简单参数
1)声明查询接口
```java
@EsQueryIndex(value = "person_es_index")
public interface PersonEsParamRepository extends BaseESRepository {
/**
* List查询
*
* @return
*/
List queryList(@Terms List personNoList);
}
```
2)测试示例
```java
@Slf4j
@RunWith(SpringRunner.class)
@SpringBootTest
public class EsEngineProxyModelQueryTest {
@Resource
private PersonEsParamRepository personEsParamRepository;
/**
* List查询测试
*/
@Test
public void queryListResponse() {
List personNoList = Lists.newArrayList("US2022060100001", "US2022060100002");
List res = personEsParamRepository.queryList(personNoList);
log.info("res:{}", JsonParser.asJson(res));
}
}
```
3) 查询效果
```json
{
"size": 1000,
"timeout": "10s",
"query": {
"bool": {
"filter": [
{
"terms": {
"personNo": [
"US2022060100001",
"US2022060100002"
]
}
}
]
}
}
}
```
### 2.sql查询
1)声明查询接口
```java
@EsQueryIndex("person_es_index")
public interface PersonEsSqlRepository extends BaseEsRepository {
/**
* 对象参数测试
* @param person
* @return
*/
@EsQuery("SELECT * FROM person_es_index WHERE status = #{person.status} AND sex = #{person.sex}")
List pageQuery(PersonEntity person);
}
```
2)测试示例
```java
/**
* 对象参数查询 测试
*/
@Slf4j
@RunWith(SpringRunner.class)
@SpringBootTest
public class EsEngineProxySqlQueryTest {
@Resource
private PersonEsSqlRepository personEsSqlRepository;
@Test
public void testSqlPageQuery() {
PersonEntity person = new PersonEntity();
person.setStatus(1);
person.setSex(1);
List results = personEsSqlRepository.pageQuery(person);
System.out.println(JsonParser.asJson(results));
}
}
```
3)查询效果
```
2022-06-21 15:46:24.781INFO 52845---[main]c.e.e.b.c.q.sql.EsSqlExecuteHandler:http://localhost:9200/_sql?format=json
2022-06-21 15:46:24.781INFO 52845---[main]c.e.e.b.c.q.sql.EsSqlExecuteHandler:{"query":"SELECT * FROM person_es_index WHERE status = 1 AND sex = 1"}
```
### 3.扩展查询
#### 3.1 扩展查询说明
##### 3.1.1扩展查询原理
###### 1)普通查询
拦截orm框架执行过程中生成的sql, 对sql进行改写后, 查询es返回结果
###### 2)回表查询
拦截orm框架执行过程中生成的sql, 对sql进行改写后, 查询es返回唯一索引,通过唯一索引查询 mysql返回明细
##### 3.1.1 sql改写规则
①替换表名为es索引名
②清除关联查询
③清除from,where,group by,having,order by 中的表别名(t.xx,d.xx)
#### 3.2 扩展查询示例
##### 3.2.1 mybatis
1)添加maven依赖
```xml
com.elasticsearch.engine
elasticsearch-engine-mybatis
0.0.1-SNAPSHOT
```
2)mapper接口添加对应的es查询注解
```java
@EsQueryIndex("person_es_index")
@Mapper
public interface PersonMapper {
@MybatisEsQuery
PersonEsEntity queryOne(@Param("personNo") String personNo, @Param("status") Integer status);
}
```
3)测试示例
```java
@Slf4j
@RunWith(SpringRunner.class)
@SpringBootTest
public class EsEngineExtendMybatisQueryTest {
@Resource
private PersonMapper personMapper;
/**
* 单个查询
*/
@Test
public void testSqlOne() {
PersonEsEntity personEsEntity = personMapper.queryOne("US2022060100001", 1);
log.info("res:{}", JsonParser.asJson(personEsEntity));
}
}
```
4)查询效果
```
2022-06-21 15:54:48.017 INFO 53454 --- [ main] c.e.e.m.i.MybatisEsQueryInterceptor : 原始sql: SELECT * FROM person WHERE person_no = ? AND status = ?
2022-06-21 15:54:48.075 INFO 53454 --- [ main] c.e.e.m.i.MybatisEsQueryInterceptor : 改写后sql: SELECT * FROM person_es_index WHERE personNo = ? AND status = ?
2022-06-21 15:54:48.076 INFO 53454 --- [ main] c.e.e.m.i.MybatisEsQueryInterceptor : 替换参数后sql: SELECT * FROM person_es_index WHERE personNo = 'US2022060100001' AND status = 1
2022-06-21 15:54:48.076 INFO 53454 --- [ main] c.e.e.b.c.q.sql.EsSqlExecuteHandler : http://localhost:9200/_sql?format=json
2022-06-21 15:54:48.076 INFO 53454 --- [ main] c.e.e.b.c.q.sql.EsSqlExecuteHandler : {"query":"SELECT * FROM person_es_index WHERE personNo = 'US2022060100001' AND status = 1"}
```
##### 3.2.2 jpa
1)添加maven依赖
```xml
com.elasticsearch.engine
elasticsearch-engine-jpa
0.0.1-SNAPSHOT
```
2)repository接口添加对应的es查询注解
```java
@EsQueryIndex("person_es_index")
public interface PersonRepository extends JpaRepository {
@JpaEsQuery
PersonEntity getByPersonNoAndStatus(String personNo, Integer status);
}
```
3)测试示例
```java
@Slf4j
@RunWith(SpringRunner.class)
@SpringBootTest
public class EsEngineExtendJpaQueryTest {
@Resource
private PersonRepository personRepository;
/**
* 单个查询
*/
@Test
public void testSqlOne() {
PersonEntity personEntity = personRepository.getByPersonNoAndStatus("US2022060100001", 1);
log.info("res:{}", JsonParser.asJson(personEntity));
}
}
```
4)查询效果
```
2022-06-21 16:00:20.962 INFO 53773 --- [ main] c.e.e.b.c.parse.sql.EsSqlQueryHelper : 原始sql: select personenti0_.id as id1_1_, personenti0_.address as address2_1_, personenti0_.company as company3_1_, personenti0_.create_time as create_t4_1_, personenti0_.create_user as create_u5_1_, personenti0_.person_name as person_n6_1_, personenti0_.person_no as person_n7_1_, personenti0_.phone as phone8_1_, personenti0_.salary as salary9_1_, personenti0_.sex as sex10_1_, personenti0_.status as status11_1_ from person personenti0_ where personenti0_.person_no='US2022060100001' and personenti0_.status=1
2022-06-21 16:00:21.008 INFO 53773 --- [ main] c.e.e.b.c.parse.sql.EsSqlQueryHelper : 改写后sql: SELECT id, address, company, createTime, createUser, personName, personNo, phone, salary, sex, status FROM person_es_index WHERE personNo = 'US2022060100001' AND status = 1
2022-06-21 16:00:21.009 INFO 53773 --- [ main] c.e.e.b.c.parse.sql.EsSqlQueryHelper : 替换参数后sql: SELECT id, address, company, createTime, createUser, personName, personNo, phone, salary, sex, status FROM person_es_index WHERE personNo = 'US2022060100001' AND status = 1
2022-06-21 16:00:21.010 INFO 53773 --- [ main] c.e.e.b.c.q.sql.EsSqlExecuteHandler : http://localhost:9200/_sql?format=json
2022-06-21 16:00:21.010 INFO 53773 --- [ main] c.e.e.b.c.q.sql.EsSqlExecuteHandler : {"query":"SELECT id, address, company, createTime, createUser, personName, personNo, phone, salary, sex, status FROM person_es_index WHERE personNo = 'US2022060100001' AND status = 1"}
```
##### 3.2.3 jooq
1)添加maven依赖
```xml
com.elasticsearch.engine
elasticsearch-engine-jooq
0.0.1-SNAPSHOT
```
2)dao实现类添加对应的es查询注解
```java
@EsQueryIndex("person_es_index")
@Component
public class PersonJooqDaoImpl implements PersonJooqDao {
@Autowired
private DSLContext context;
private final Person PERSON = Tables.PERSON;
/**
* @param personNo
* @param status
* @return
*/
@JooqEsQuery
@Override
public PersonEntity getByPersonNoAndStatus(String personNo, Integer status) {
return context.selectFrom(PERSON).where(
PERSON.PERSON_NO.eq(personNo).and(PERSON.STATUS.eq(status.byteValue()))
).fetchOneInto(PersonEntity.class);
}
}
```
3)测试示例
```java
@Slf4j
@RunWith(SpringRunner.class)
@SpringBootTest
public class EsEngineExtendJooqQueryTest {
@Resource
private PersonJooqDao personJooqDao;
/**
* 单个查询
*/
@Test
public void testSqlOne() {
PersonEntity personEntity = personJooqDao.getByPersonNoAndStatus("US2022060100001", 4);
log.info("res:{}", JsonParser.asJson(personEntity));
}
}
```
4)查询效果
```
2022-06-21 16:03:05.629 INFO 53945 --- [ main] c.e.e.b.c.parse.sql.EsSqlQueryHelper : 原始sql: select
`user`.`person`.`id`,
`user`.`person`.`person_no`,
`user`.`person`.`person_name`,
`user`.`person`.`phone`,
`user`.`person`.`salary`,
`user`.`person`.`company`,
`user`.`person`.`status`,
`user`.`person`.`sex`,
`user`.`person`.`address`,
`user`.`person`.`create_time`,
`user`.`person`.`create_user`
from `user`.`person`
where (
`user`.`person`.`person_no` = 'US2022060100001'
and `user`.`person`.`status` = 4
)
2022-06-21 16:03:05.674 INFO 53945 --- [ main] c.e.e.b.c.parse.sql.EsSqlQueryHelper : 改写后sql: SELECT `id`, `personNo`, `personName`, `phone`, `salary`, `company`, `status`, `sex`, `address`, `createTime`, `createUser` FROM person_es_index WHERE (`personNo` = 'US2022060100001' AND `status` = 4)
2022-06-21 16:03:05.675 INFO 53945 --- [ main] c.e.e.b.c.parse.sql.EsSqlQueryHelper : 替换参数后sql: SELECT id, personNo, personName, phone, salary, company, status, sex, address, createTime, createUser FROM person_es_index WHERE (personNo = 'US2022060100001' AND status = 4)
2022-06-21 16:03:05.676 INFO 53945 --- [ main] c.e.e.b.c.q.sql.EsSqlExecuteHandler : http://localhost:9200/_sql?format=json
2022-06-21 16:03:05.676 INFO 53945 --- [ main] c.e.e.b.c.q.sql.EsSqlExecuteHandler : {"query":"SELECT id, personNo, personName, phone, salary, company, status, sex, address, createTime, createUser FROM person_es_index WHERE (personNo = 'US2022060100001' AND status = 4)"}
```
##### 3.2.4 关联查询(以mybatis为例)
###### 3.2.4.1 关联查询说明
1)应用场景
elasticsearch 存储的字段为mysql多张表聚合的字段,mysql 原本的查询为关联多表查询
###### 3.2.4.2 关联查询示例
1)mapper接口添加对应的es查询注解
```java
@EsQueryIndex("person_es_index")
@Mapper
public interface PersonExtendMapper {
int insertList(List persons);
@MybatisEsQuery
List queryList(@Param("status") Integer status, @Param("hobby")String hobby);
}
```
3)测试示例
```java
@Slf4j
@RunWith(SpringRunner.class)
@SpringBootTest
public class EsEngineExtendMybatisQueryTest {
@Resource
private PersonExtendMapper personExtendMapper;
/**
* 关联查询测试
*/
@Test
public void testJoinQueryList() {
List results = personExtendMapper.queryList(4,"踢足球");
System.out.println(JsonParser.asJson(results));
}
}
```
4)查询效果
```
2022-06-23 00:23:00.012 INFO 37281 --- [ main] c.e.e.m.i.MybatisEsQueryInterceptor : 原始sql: SELECT * FROM person p INNER JOIN person_extend pe
ON p.person_no = pe.person_no
WHERE p.status = ? AND pe.hobby=?
2022-06-23 00:23:00.052 INFO 37281 --- [ main] c.e.e.m.i.MybatisEsQueryInterceptor : 改写后sql: SELECT * FROM person_es_index WHERE status = ? AND hobby = ?
2022-06-23 00:23:00.053 INFO 37281 --- [ main] c.e.e.m.i.MybatisEsQueryInterceptor : 替换参数后sql: SELECT * FROM person_es_index WHERE status = 4 AND hobby = '踢足球'
2022-06-23 00:23:00.054 INFO 37281 --- [ main] c.e.e.b.c.q.sql.EsSqlExecuteHandler : http://localhost:9200/_sql?format=json
2022-06-23 00:23:00.054 INFO 37281 --- [ main] c.e.e.b.c.q.sql.EsSqlExecuteHandler : {"query":"SELECT * FROM person_es_index WHERE status = 4 AND hobby = '踢足球'"}
```
##### 3.2.5 回表查询(以mybatis为例)
###### 3.2.5.1 回表查询说明
1)应用场景
① elasticsearch 存储的非全量字段,而只有搜索字段, 通过es搜索唯一索引后,再用唯一索引回表查询mysql
② elasticsearch 存在延迟,通过es搜索出es搜索唯一索引后,再用唯一索引回表查询mysql
2)sql改写规则
① es执行的sql,再原改写的基础上 改写查询字段仅查询回表字段
② 回表sql, 再原orm框架sql基础上拼接 es执行结果的回表查询条件
###### 3.2.5.2 回表查询示例
1)mapper接口添加对应的es查询注解
```java
@EsQueryIndex("person_es_index")
@Mapper
public interface PersonMapper {
@MybatisEsQuery(backColumn = "id",backColumnType = Long.class)
List findBySex(@Param("sex") Integer sex);
}
```
3)测试示例
```java
@Slf4j
@RunWith(SpringRunner.class)
@SpringBootTest
public class EsEngineExtendMybatisQueryTest {
@Resource
private PersonMapper personMapper;
/**
* 回表查询测试 id
*/
@Test
public void testSqlBackById() {
List results = personMapper.findBySex(1);
System.out.println(JsonParser.asJson(results));
}
}
```
4)查询效果
```
2022-06-22 00:46:23.302 INFO 7723 --- [ main] c.e.e.m.i.MybatisEsQueryInterceptor : 原始sql: SELECT * FROM person WHERE sex = ?
2022-06-22 00:46:23.347 INFO 7723 --- [ main] c.e.e.m.i.MybatisEsQueryInterceptor : 改写后sql: SELECT id FROM person_es_index WHERE sex = ?
2022-06-22 00:46:23.348 INFO 7723 --- [ main] c.e.e.m.i.MybatisEsQueryInterceptor : 替换参数后sql: SELECT id FROM person_es_index WHERE sex = 1
2022-06-22 00:46:23.349 INFO 7723 --- [ main] c.e.e.b.c.q.sql.EsSqlExecuteHandler : http://localhost:9200/_sql?format=json
2022-06-22 00:46:23.349 INFO 7723 --- [ main] c.e.e.b.c.q.sql.EsSqlExecuteHandler : {"query":"SELECT id FROM person_es_index WHERE sex = 1"}
2022-06-22 00:46:24.480 INFO 7723 --- [ main] c.e.e.m.i.MybatisEsQueryInterceptor : 回表sql : SELECT * FROM person WHERE sex = ? AND id IN (7, 13, 17, 6, 9, 14, 16, 23, 24)
```
## 使用示例
https://gitee.com/my-source-project/elasticsearch-engine-demo
https://github.com/wanghuan9/elasticsearch-engine-demo
## 相关文档
待补全...
## 兼容性
elasticsearch 字段命名支持 驼峰和下划线
elasticsearch 版本支持 v6 和 v7
## 参考及引用
本项目 注解查询参考了 开源项目 https://gitee.com/JohenTeng/elasticsearch-helper (感谢大佬)
## 标签
elasticsearchsql, elasticsearchjdbc,elasticsearchmybatis,elasticsearchjpa,elasticsearchjooq,
elasticsearchquery,elasticsearch查询,elasticsearch查询引擎,elasticsearch查询工具, Elasticsearchapi,Elasticsearchclient,Use SQL to
query Elasticsearch,
sql拦截,sql拦截器, jpasql拦截,mybatissql拦截,Mybatis拦截器,jpa拦截器,jooq拦截器,
jooqsql拦截,sql拦截参数填充,Interceptor,StatementInspector,DefaultExecuteListener
hibernate 拦截sql ,输出sql语句, 获取sql语句,JPA 打印原生sql,输出真实的sql语句,输出mybatis完整SQL语句,输出jpa完整SQL语句,输出jooq完整SQL语句
jsqlparser,sql解析器,向sql语句中插入where条件,mybatis拦截器,修改sql,JSqlParser,JsqlParser插件用来对于SQL语句进行解析和组装,完整的SQL语句打印,打印完整SQL语句(无问号)
,Hibernate拦截器
,PlainSelect.setFromItem