代码拉取完成，页面将自动刷新
Tree Sitter Multi Codeview Generator aims to generate combined multi-code view graphs that can be used with various types of machine learning models (sequence models, graph neural networks, etc).
comex is a rebuild of Tree Sitter Multi Codeview Generator for easier invocation as a Python package. This rebuild also includes a cli interface. Currently, comex generates codeviews for Java and C#, for both method-level and file-level code snippets. comex can be used to generate over $15$ possible combinations of codeviews for both languages (complete list here). comex is designed to be easily extendable to various programming languages. This is primarliy because we use tree-sitter for parsing, a highly efficient incremental parser that supports over $40$ languages. If you wish to add support for more languages, please refer to the contributing guide.
If you wish to learn more about the approach taken, here are some conference talks and publications:
Comex played a critical role in developing CodeSAM, a novel framework for infusing multiple code-views into transformer-based models. CodeSAM builds on tools like comex to create structured code representations (e.g., AST, CFG, DFG), enabling fine-tuning of language models like CodeBERT. Experimental results show that by using this technique, we can improve downstream performance when compared to SLMs like GraphCodeBERT and CodeBERT on ML for SE tasks by utilizing individual code-views or a combination of code-views during fine-tuning.
You can find more details about CodeSAM in the following pre-print:
If you use Comex in your research, please cite our work by using the following BibTeX entry:
@inproceedings{das2023comex,
title={COMEX: A Tool for Generating Customized Source Code Representations},
author={Das, Debeshee and Mathews, Noble Saji and Mathai, Alex and Tamilselvam, Srikanth and Sedamaki, Kranthi and Chimalakonda, Sridhar and Kumar, Atul},
booktitle={2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)},
pages={2054--2057},
year={2023},
organization={IEEE}
}
comex is published on the Python Registry and can be easily installed via pip:
pip install comex
Note: You would need to install GraphViz(dot) so that the graph visualizations are generated
To setup comex for development using the source code in your python environment:
pip install -r requirements-dev.txt
This performs an editable install, meaning that comex would be available throughout your environment (particularly relevant if you use conda or something of the sort). This means now you can interact and import from comex just like any other package while remaining standalone but also reflecting any code side updates without any other manual steps
This is the recommended way to get started with comex as it is the most user friendly
The attributes and options supported by the CLI are well documented and can be viewed by running:
comex --help
For example, to generate a combined CFG and DFG graph for a java file, you can run:
comex --lang "java" --code-file ./test.java --graphs "cfg,dfg"
The comex package can be used by importing required drivers as follows:
from comex.codeviews.combined_graph.combined_driver import CombinedDriver
CombinedDriver(
src_language=lang,
src_code=code,
output_file="output.json",
graph_format=output,
codeviews=codeviews
)
In most cases the required combination can be obtained via the combined_driver module as shown above.
src_language: denotes one of the supported languaged hence currently "java" or "cs"
src_code: denotes the source code to be parsed
output_file: denotes the output file to which the generated graph is written
graph_format: denotes the format of the output graph. Currently supported formats are "dot" and "json". To generate both pass "all"
codeviews: refers to the configuration passed for each codeview
While comex provides method-level and file-level support for both Java and C#, it's important to note the following limitations and known issues:
No Inter-file Analysis Support: The tool currently does not support codeviews that involve interactions between multiple Java files. It is designed to generate codeviews for individual Java files only.
Syntax Errors in Code: Despite supporting non-compileable code, to ensure accurate codeviews, the input Java code must be free of syntax errors. Code with syntax errors may not be correctly parsed and displayed in the generated codeviews.
Limited Support for Function Call Arguments: The tool does not provide proper support for when a function call is passed as an argument to another function call in Java code. The resulting codeview might not accurately represent the intended behavior in such cases.
In addition to the limitations mentioned for Java, the tool has the following limitations specific to C#:
No Support for Lambda Functions and Arrow Expressions: The tool does not support codeviews involving lambda functions and arrow expressions in C#. The generated codeviews may not accurately represent these language features.
No Support for Compiler Directives: Compiler directives, such as pragma directives, are not supported by the tool. Code involving such directives may not be properly displayed in the generated codeviews.
Incomplete Operator Declaration Support: The tool may have limited support for operator declarations in C#. Certain constraints and edge cases related to operator overloading might not be fully captured in the generated codeviews.
Limited Support for Inheritance and Abstraction: The tool's support for inheritance and abstraction in C# is not fully comprehensive. Codeviews involving complex inheritance hierarchies or advanced abstraction patterns may not be accurately represented.
Please note that while we continuously work to improve the tool and address these limitations, the current implementation may not be perfect. We appreciate your understanding and encourage you to provide feedback and report any issues you encounter, as this helps us enhance the tool's capabilities.
Combined simple AST+CFG+DFG for a simple Java program that finds the maximum among 2 numbers:
Below we present more examples of input code snippets and generated codeviews for both Java and C#.
CLI Command:
comex --lang "java" --code-file sample/example.java --graphs "cfg,dfg"
Java Code Snippet:
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;
public class Main {
// static String INPUT = "5\n3 2 2 4 1\n1 2 2 2 1";
static String INPUT = "";
public static void main(String[] args) {
InputStream is = INPUT.isEmpty() ? System.in : new ByteArrayInputStream(INPUT.getBytes());
Scanner scanner = new Scanner(is);
final int n = scanner.nextInt();
List<Position> positionList = new ArrayList<>(n);
for (int i = 0; i < n; i++) {
positionList.add(
new Position(
scanner.nextInt(),
scanner.nextInt(),
scanner.nextInt()
)
);
}
System.out.println(solve(positionList) ? "Yes" : "No");
}
static class Position {
int t;
int x;
int y;
public Position(int t, int x, int y) {
this.t = t;
this.x = x;
this.y = y;
}
}
static boolean solve(List<Position> positionList) {
Position currentPosition = new Position(0, 0, 0);
for (int i = 0; i < positionList.size(); i++) {
Position nextPosition = positionList.get(i);
if (!possibleMove(currentPosition.t, nextPosition.t, currentPosition.x, nextPosition.x, currentPosition.y, nextPosition.y)) {
return false;
}
currentPosition = nextPosition;
}
return true;
}
static boolean possibleMove(int t1, int t2, int x1, int x2, int y1, int y2) {
int tDiff = t2 - t1;
int absX = Math.abs(x1 - x2);
int absY = Math.abs(y1 - y2);
if (absX + absY <= tDiff) {
if (tDiff % 2 == (absX + absY) % 2) {
return true;
}
}
return false;
}
}
Generated Codeview:
CLI Command:
comex --lang "cs" --code-file sample/example.cs --graphs "cfg,dfg"
C# Code Snippet:
public class DFG_A2 {
public void main(string[] args) {
BufferedReader br = new BufferedReader(new InputStreamReader(System.in)); // {}
String str = br.attribute1; // {3}
str = br.attribute2.method1(); // {3}
br.attribute1 = br.attribute2; // {3,5}
br.method2(br.attribute1, br.attribute2); // {3,5,6}
BufferedReader br2 = br; // {5,6,7}
br.method3(); // {5,6,7}
int j = br2.attribute1.method2(3,4); // {8}
}
}
Generated Codeview:
More examples and results can be found in the tests/data directory
The code is structured in the following way:
The repo is setup to automatically perform CI tests on making pulls to main and development branches. To test locally:
Run specific test
test_cfg function'[...]' part to run all tests in a file
pytest -k 'test_cfg[cs-test7]' --no-cov
Run all tests and get coverage report
pytest
Analyze the deviation report given by deepdiff by using the verbose output.
This will help quickly figure out difference from the gold file
pytest -k 'test_cfg[cs-test7]' --no-cov -vv
Make sure to bump the version in setup.cfg.
Then run the following commands:
rm -rf build dist
python setup.py sdist bdist_wheel
Then upload it to PyPI using twine (pip install twine if not installed):
twine upload dist/*
This tool was developed for research purposes as a part of the OSCP Project. Efficient representation of source code is essential for various software engineering tasks using AI pipelines such as code translation, code search and code clone detection. Code Representation aims at extracting the both syntactic and semantic features of source code and representing them by a vector which can be readily used for the downstream tasks. Multiple works exist that attempt to encode the code as sequential data to easily leverage state of art NN models like transformers. But it leads to a loss of information. Graphs are a natural representation for the code but very few works(MVG-AAAI’22) have tried to represent the different code features obtained from different code views like Program Dependency Graph, Data Flow Graph etc. as a multi-view graph. In this work, we want to explore more code views and its relevance to different code tasks as well as leverage transformers model for the multi-code view graphs. We believe such a work will help to
This tool is based on the ongoing joint research effort between IBM and Risha Lab at IIT Tirupati to explore the effects of different code representations on code based tasks involving:
此处可能存在不合适展示的内容，页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容，可点击提交进行申诉，我们将尽快为您处理。
马建仓 AI 助手