关于CSharp版本连接器封装改进建议

关于CSharp版本连接器封装改进建议

鄙人近期工作开始着手TDengineDB时序数据库的相关技术探索；我司目前已选定该数据库作为物联网相关设备的数据存储，每设备所产生的数据量，在1秒内通常会大于等于50条上下；我司在单客户所部署的边缘端设备，一般不低于50台，且均为7*24小时作业，所以对于数据存储的压力和效率尤为重视。

考虑使用CSharp来作为数据存取开发工作，也是经过了整体产品各环节需求确定的。目前，在使用TDengineDB原生C#连接器时发现，其基本就是一个基于C导出函数的C#导入函数类；同时，发现使用了大量的IntPtr这种非托管资源访问；对于具备一定托管与非托管资源操作的开发人员，在明白函数定义时，使用起来还可以；但当对于非托管资源不是很熟悉的开发人员，则是个很大的负担，特别是操作不当，直接导致程序崩溃，而且最怕就是最直接的底层崩溃，而C#里面是无法捕获到的。

因此，我马上想到了之前研究过的SQLite数据库，微软本身也对SQLite做了很友好的支持；微软最新封装的Microsoft.Data.Sqlite里面改善了很多，它是基于SQLitePCLRaw库进行的再封装；SQLitePCLRaw库则是基于SQLite原生C API的CSharp封装，可以作为不错的参考，或者其它关于C与C#互操作的封装。

我近期为公司的一个C API库进行了CSharp的导入函数封装工作，SQLitePCLRaw对于C API采用private访问修饰，我目前封装的采用internal访问修饰，后期我考虑也借鉴SQLitePCLRaw的封装思想。我目前的封装形式比较简单，因为C API库还在开发阶段，只是进行了简单的限制和提供对外友好访问；封装的目的是尽可以减少外部感知到IntPtr等非托管资源，以及内部尽可能消化unsafe之类的定义，让外部调用看起来跟使用C#类一样。

我想我们TDengineDB的连接器也是一样的原理，可以考虑更多层面的工作，让我们的连接器更丰满；目前.net已经来到了6.0时代，随之而来将是7.0等后续版本，微软在新版本中对于托管与非托管资源的交互作了很大的改进，我们是不是也可以吸收一些相关的技术，运用到我们的连接器中来？特别是异步编程的支持等。

末了，我初步阅读了TDengineDB相关文档，其中有建议的数据插入时，尽量进行批量插入的操作，能使TDengineDB使用更高、更快、更强、更团结……好像跑题了！但是，通常的应用场景，有可能是收到一条消息后立即进行入库；或者，产生一条数据入库一条数据；此时，可能不便将多条数据进行打包，然后再同批次insert入库。

此时，可能会说：在业务端先缓存起来呀，然后缓存一定数量后再批量入库！是的，这样是可行的，而且还有更加多的方法、方式等，但对于使用方来说，则需要本身因此而编写大量的逻辑处理等等，有可能会存在很多的判断操作，这些综合在一起，肯定会影响到整体的效率！

因此，像这样的需求肯定不止这一种，而通常使用连接器等的开发人员，对于我们TDengineDB数据库而言不是最最最了解的，可能有很多的优化方法不知道如何使用；那么，如何我们本身的实现人员，基本一些使用优化，再运用到连接器（SDK）的开发中去，是不是更好呢？

最后，时间有些晚了，很久没有打这么多字了，上下班路途用时有些长，已经晚睡一个小时了，为了不影响明天工作状态，我就不校对上面的文字了，如果有哪里不当，还望见谅！鄙人粗鄙之言，切勿介怀！

祝 TDengineDB 越来越好，为国争光！一起向未来(Together for a Shared Future)。

Suggestions on the improvement of CSharp version connector package

I have recently started to explore the technology related to TDengineDB timing database; our company has selected this database as the data storage for IoT-related devices, and the amount of data generated by each device is usually greater than or equal to 50 items in 1 second; the edge-end devices deployed by our company in a single customer are generally not less than 50 units, and they are all operated 7*24 hours, so the pressure and efficiency of data Therefore, the pressure and efficiency of data storage are particularly important.

The consideration of using CSharp for data access development work is also determined after the overall product requirements of each link. At present, when using TDengineDB native C# connector, it is found that it is basically a C# import function class based on C export function; at the same time, it is found that a large number of unmanaged resources such as IntPtr are used for access; for developers who have certain managed and unmanaged resource operations, it is OK to use when they understand the function definition; however, when developers who are not very familiar with unmanaged resources, it is a big problem. But when the developers who are not very familiar with unmanaged resources, it is a great burden, especially if the operation is not done properly, which directly leads to the program crash, and the most fear is the most direct underlying crash, which can not be captured in C#.

Therefore, I immediately thought of the SQLite database that I had studied before, and Microsoft itself has made a very friendly support for SQLite; Microsoft's latest package Microsoft. SQLitePCLRaw library is based on the native C API CSharp package, which can be a good reference, or other packages about C and C# interoperability.

SQLitePCLRaw uses private access modifier for C API, and my current package uses internal access modifier, and I will consider drawing on SQLitePCLRaw's package idea later. My current form of encapsulation is relatively simple, because the C API library is still in the development stage, only simple restrictions and provide external friendly access; the purpose of encapsulation is to reduce external perception of unmanaged resources such as IntPtr as much as possible, as well as internal digestion of unsafe and other definitions, so that external calls look the same as using C# classes.

I think our TDengineDB connector is the same principle, and we can consider more levels of work to make our connector more fleshed out; at present, .net has come to the era of 6.0, followed by 7.0 and other subsequent versions, and Microsoft has made great improvements to the interaction between managed and unmanaged resources in the new version, so can we also absorb some relevant technologies and apply them to our connector? our connectors? In particular, support for asynchronous programming, etc.

At the end, I initially read the TDengineDB related documents, where there are recommendations for data insertion, try to perform bulk insertion operations, which can make TDengineDB use higher, faster, stronger and more united ...... seems to be off-topic! However, the usual application scenario, it is possible to receive a message immediately after the library; or, produce a data into a data library; at this time, it may be inconvenient to package multiple data, and then the same batch INSERT into the library.

At this point, you may say: in the business side of the first cache up ah, and then cache a certain number of batches into the library! Yes, this is feasible, and there are more methods, ways, etc., but for the user, it is necessary to write a lot of logic processing and so on, there may be a lot of judgment operations, these combined, will certainly affect the overall efficiency!

Therefore, there must be more than one demand like this, and developers who usually use connectors and so on are not the most most knowledgeable about our TDengineDB database, and may have many optimization methods that they do not know how to use; then, how can our own implementation staff, some basic use optimization, and then apply it to the development of connectors (SDK), is it better?

Finally, the time is a little late, it has been a long time since I typed so many words, the commute to and from work took a little longer, has slept an hour late, in order not to affect the state of work tomorrow, I will not proofread the above text, if there where improper, please forgive me! I'm sorry if there's anything wrong!

I wish TDengineDB to be better and better, and to win the honor of our country! Together to the future.

Translated with www.DeepL.com/Translator (free version)

GVP taosdata/TDengine

内容风险标识

评论 (0)

GVPtaosdata/TDengine

内容风险标识