Description

Salt Inspector, formerly known as "Salt Node" is running an SQLite under the hood in order to buffer query results that usually taking time. After a while using this, the SQLite appears to be a quite overhead for this task as well as asking for an additional maintainable dependency.

Goals

The idea is to remove it and replace with some sort of simple object storage that mainly does only two things efficiently:

1. Stores objects
2. Retrieves objects
3. (2a) Does update/delete but within "nice to have" scope (can be inefficient as not needed for many objects)

Something else?

Additionally, this cache could:

1. Store data in some sort of a plain text
2. Provide an easy fast intersection between the versions for retrieving a difference

More?

Would it also:

1. Shrink the code a bit?
2. Remove those u̶g̶l̶y̶ not beautiful SQL queries from within the code?
3. Work only with Python objects? Directly?
4. Have a very small codebase?

Results:

1. Storage can serialize a Python objects and store it to the CSV file (comma separated values) and work on it while it is compressed (GNU Zip)
2. Storage get retrieve a Python objects from such CSV file
3. Data is smaller than would be stored in a binary form (msgpack for example). The binary form storing would require object-per-file (bad idea) or maintain own binary format to read records (bad idea again).
4. Data can be processed with any other tools (sed or awk etc).
5. Is as fast as before.
6. Storage is based on time-proven CSV module, which is since Python 2.3 or 13 years old, and is widely used since.
7. Renamed from node to inspector. Finally... Sorry, node.js guys. :-)
8. "YES" to all those questions in the section "Would it also?.."
9. Unit tests are written.

Bonus features:

1. Treating each inspection full scanning as a snapshot. For this implemented listing and deleting of such snapshots.
2. Started working on diff'ing data between the snapshots (not upstream yet).

Pull request on Github/Salstack is here.