# DataFrame
**Repository Path**: scientific-speculation/DataFrame
## Basic Information
- **Project Name**: DataFrame
- **Description**: C++ DataFrame for statistical, Financial, and ML analysis -- in modern C++ using native types, continuous memory storage, and no pointers are involved
- **Primary Language**: Unknown
- **License**: BSD-3-Clause
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No
## Statistics
- **Stars**: 2
- **Forks**: 0
- **Created**: 2021-01-09
- **Last Updated**: 2022-01-18
## Categories & Tags
**Categories**: Uncategorized
**Tags**: None
## README
[](https://ci.appveyor.com/project/hosseinmoein/dataframe)
[](https://travis-ci.org/hosseinmoein/DataFrame)
[](https://isocpp.org/std/the-standard )
[](https://app.codacy.com/manual/hosseinmoein/DataFrame?utm_source=github.com&utm_medium=referral&utm_content=hosseinmoein/DataFrame&utm_campaign=Badge_Grade_Dashboard)
[](https://mybinder.org/v2/gh/hosseinmoein/DataFrame/master)
# DataFrame
This is a C++ statistical library that provides an interface similar to Pandas package in Python.
A DataFrame can have one index column and many data columns of any built-in or user-defined type.
You could slice the data in many different ways. You could join, merge, group-by the data. You could run various statistical, summarization and ML algorithms on the data. You could add your custom algorithms easily. You could multi-column sort, custom pick and delete the data. And more …
DataFrame also includes a large collection of analytical routines in form of visitors -- see [documentation](https://htmlpreview.github.io/?https://github.com/hosseinmoein/DataFrame/blob/master/docs/HTML/DataFrame.html) below. These are from basic stats such as Mean, Std Deviation, Return, … to more involved analysis such as Affinity Propagation, Polynomial Fit, Hurst Exponent, … -- See [documentation](https://htmlpreview.github.io/?https://github.com/hosseinmoein/DataFrame/blob/master/docs/HTML/DataFrame.html) for a complete list with code samples, and how you can add your custom algorithms.
I have followed a few principles in this library:
1. Support any type either built-in or user defined without needing new code
2. Never chase pointers ala `linked lists`, `std::any`, `pointer to base`, ..., including `virtual function calls`
3. Have all column data in continuous memory space. Also, be mindful of cache-line aliasing misses between multiple columns
4. Never use more space than you need ala `unions`, `std::variant`, ...
5. Avoid copying data as much as possible
6. Use multi-threading but only when it makes sense
7. Do not attempt to protect the user against `garbage in, garbage out`
Views
You can slice the data frame and instead of getting another data frame you can opt to get a view. A view is a data frame that is a reference to a slice of the original data frame. So if you change the data in the view the corresponding data in the original data frame will also be changed (and vice versa).
Multithreading
1. DataFrame uses static containers to achieve type heterogeneity. By default, these static containers are unprotected. This is done by design. So by default, there is no locking overhead. If you use DataFrame in a multithreaded program you must provide a _SpinLock_ defined in ThreadGranularity.h file. DataFrame will use your _SpinLock_ to protect the containers.
Please see documentation, set_lock(), remove_lock(), and dataframe_tester.cc for code example.
2. In addition, instances of DataFrame are not multithreaded safe either. In other words, a single instance of DataFrame must not be used in multiple threads without protection, unless it is used as read-only.
3. In the meantime, DataFrame utilizes multithreading in two different ways internally:
1. Async Interface: There are asynchronous versions of some methods. For example, you have _sort()_/_sort_async()_, _visit()_/_visit_async()_, ... more. The latter versions return a _std::future_ that could execute in parallel.
2. DataFrame uses multiple threads, internally and unbeknown to the user, in some of its algorithms when appropriate. User can control (or turn off) the multithreading by calling _set_thread_level()_ which sets the max number of threads to be used. The default is 0. The optimal number of threads is a function of users hardware/software environment and usually obtained by trail and error. _set_thread_level()_ and threading level in general is a static property and once set, it applies to all instances.
DateTime
DateTime class included in this library is a very cool and handy object to manipulate date/time with nanosecond precision.
---
## [*DataFrame Documentation / Code Samples*](https://htmlpreview.github.io/?https://github.com/hosseinmoein/DataFrame/blob/master/docs/HTML/DataFrame.html)
[DateTime Documentation](docs/DateTimeDoc.pdf)
---
[DataFrame Test File](test/dataframe_tester.cc)
[DataFrame Test File 2](test/dataframe_tester_2.cc)
[Heterogeneous Vectors Test File](test/vectors_tester.cc)
[Date/Time Test File](test/date_time_tester.cc)
---
[Contributions](docs/CONTRIBUTING.md)
[License](License)
---
### Installing using CMake
```bash
mkdir [Debug | Release]
cd [Debug | Release]
cmake -DCMAKE_BUILD_TYPE=[Debug | Release] ..
make
make install
```
### Uninstalling
```bash
cd [Debug | Release]
make uninstall
```
### Package managers
If you are using Conan to manage your dependencies, merely add `dataframe/x.y.z@` to your requires, where x.y.z is the release version you want to use. Conan will acquire DataFrame, build it from source in your computer, and provide CMake integration support for your projects. See the [conan docs](https://docs.conan.io/en/latest/) for more information.
Sample `conanfile.txt`:
```text
[requires]
dataframe/1.14.0@
[generators]
cmake
```
---
### Performance
There is a test program [_dataframe_performance_](test/dataframe_performance.cc) that should give you some sense of how this library performs. As a comparison, there is also a Pandas Python [_pandas_performance_](test/pandas_performance.py) script that does exactly the same thing.
_dataframe_performance.cc_ uses DataFrame async interface and is compiled with gcc compiler with -O3 flag.
_pandas_performance.py_ is ran with Pandas 1.1.0 and Python 3.7.
I ran both on my mac-book, doing the following:
1. Generate ~1.6 billion second resolution timestamps and load it into the DataFrame/Pandas as index.
2. Generate ~1.6 billion random numbers each for 3 columns with normal, log normal, and exponential distributions and load them into the DataFrame/Pandas.
3. Calculate the mean of each of the 3 columns.
Result:
```bash
MacBook> time python test/pandas_performance.py
All memory allocations are done. Calculating means ...
real 17m18.916s
user 4m47.113s
sys 5m31.901s
MacBook> time bin/Linux.GCC64/dataframe_performance
All memory allocations are done. Calculating means ...
real 5m25.641s
user 2m37.362s
sys 2m3.451s
```
The interesting part:
1. Pandas script, I believe, is entirely implemented in Numpy which is in C.
2. In case of Pandas, allocating memory + random number generation takes almost the same amount of time as calculating means.
3. In case of DataFrame 90% of the time is spent in allocating memory + random number generation.
4. You load data once, but calculate statistics many times. So DataFrame, in general, is about 17x faster than parts of Pandas that are implemented in Numpy. I leave parts of Pandas that are purely in Python to imagination.
5. Pandas process image at its peak is ~105GB. C++ DataFrame process image at its peak is ~68GB.