Comments

• 

  almost 2 years ago by jlausuch | Reply

  We will first start evaluating which DB offers more suitable solution for what we want.

  InfluxDB: (https://docs.influxdata.com/influxdb/v1.8/concepts/key_concepts/) - Main usecase of this DB is to store performance metrics over time. - It's pure timestamp based, the timestamp is created at data insertion time, which acts as the "primary key" of the table. - You don't need to define the fields of the table when creating it, the fields will be inserted when pushing data to it. - There are 2 types of fields: "field keys" and "field values". - Challenging (or impossible) to define relations between tables.

  Postgresql: (https://www.postgresqltutorial.com/) - More similar to the relational DB world. - Timestamp is just another optional field, not mandatory. - Possible to define primary keys (useful for our use case) - Possible to define relations between tables (although we don't need this).

  MySQL: - Maybe too powerful for what we want? https://www.guru99.com/postgresql-vs-mysql-difference.html

  So, from my point of view, InfluxDB is very limited and very much focused on collecting performance metrics over time (cpu usage, network, etc). Our use case is slightly different, we want to collect specific data of different nature (image size, number of packages, etc) and for each of these data values, we want to have some unique KEY which will be the image per se (arch, build, flavor, etc). Therefore, I think Postgres is a better fit for us.

  Also, an important thing to consider is how good those integrate with non-Grafana representation tools, like Chart.js.

×
Edit Comment 5999
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

* Use asterisks
* for lists

This is an [example link](http://example.com/)

This is an ![example image](http://paste.opensuse.org/view/raw/68957446)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
We will first start evaluating which DB offers more suitable solution for what we want. **InfluxDB**: (https://docs.influxdata.com/influxdb/v1.8/concepts/key_concepts/) - Main usecase of this DB is to store performance metrics over time. - It's pure timestamp based, the timestamp is created at data insertion time, which acts as the "primary key" of the table. - You don't need to define the fields of the table when creating it, the fields will be inserted when pushing data to it. - There are 2 types of fields: "field keys" and "field values". - Challenging (or impossible) to define relations between tables. **Postgresql**: (https://www.postgresqltutorial.com/) - More similar to the relational DB world. - Timestamp is just another optional field, not mandatory. - Possible to define primary keys (useful for our use case) - Possible to define relations between tables (although we don't need this). **MySQL**: - Maybe too powerful for what we want? https://www.guru99.com/postgresql-vs-mysql-difference.html So, from my point of view, InfluxDB is very limited and very much focused on collecting performance metrics over time (cpu usage, network, etc). Our use case is slightly different, we want to collect specific data of different nature (image size, number of packages, etc) and for each of these data values, we want to have some unique KEY which will be the image per se (arch, build, flavor, etc). Therefore, I think Postgres is a better fit for us. Also, an important thing to consider is how good those integrate with non-Grafana representation tools, like Chart.js.
 
×
Reply to jlausuch
We will first start evaluating which DB offers more suitable solution for what we want.
InfluxDB: (https://docs.influxdata.com/influxdb/v1.8/concepts/key_concepts/) - Main usecase of this DB is to store performance metrics over time. - It's pure timestamp based, the timestamp is created at data insertion time, which acts as the "primary key" of the table. - You don't need to define the fields of the table when creating it, the fields will be inserted when pushing data to it. - There are 2 types of fields: "field keys" and "field values". - Challenging (or impossible) to define relations between tables.
Postgresql: (https://www.postgresqltutorial.com/) - More similar to the relational DB world. - Timestamp is just another optional field, not mandatory. - Possible to define primary keys (useful for our use case) - Possible to define relations between tables (although we don't need this).
MySQL: - Maybe too powerful for what we want? https://www.guru99.com/postgresql-vs-mysql-difference.html
So, from my point of view, InfluxDB is very limited and very much focused on collecting performance metrics over time (cpu usage, network, etc). Our use case is slightly different, we want to collect specific data of different nature (image size, number of packages, etc) and for each of these data values, we want to have some unique KEY which will be the image per se (arch, build, flavor, etc). Therefore, I think Postgres is a better fit for us.
Also, an important thing to consider is how good those integrate with non-Grafana representation tools, like Chart.js.
---
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

* Use asterisks
* for lists

This is an [example link](http://example.com/)

This is an ![example image](http://paste.opensuse.org/view/raw/68957446)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
 
• 

  almost 2 years ago by jlausuch | Reply

  I have created a confluence page for the HowTo (setting up postgress, using Chart.js, ...). https://confluence.suse.com/display/~jlausuch/Hackweek+June+2022

×
Edit Comment 6011
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

* Use asterisks
* for lists

This is an [example link](http://example.com/)

This is an ![example image](http://paste.opensuse.org/view/raw/68957446)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
I have created a confluence page for the HowTo (setting up postgress, using Chart.js, ...). https://confluence.suse.com/display/~jlausuch/Hackweek+June+2022
 
×
Reply to jlausuch
I have created a confluence page for the HowTo (setting up postgress, using Chart.js, ...). https://confluence.suse.com/display/~jlausuch/Hackweek+June+2022
---
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

* Use asterisks
* for lists

This is an [example link](http://example.com/)

This is an ![example image](http://paste.opensuse.org/view/raw/68957446)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
 
• 

  almost 2 years ago by jlausuch | Reply

  I managed to have postgres up and running in a container with some security access to push data to it. The problem of postgres is that it doesn't come with rest api by default like for influxdb. Since we don't want to install tools in client side (openQA) like psql, we need to have some REST API for postgres. The solution is PostgREST: https://postgrest.org/en/stable/ Which is basically a proxy between the DB and the remote http requests. It offers a simple api to get and push data with simple curl command.

  In the confluence page I am adding all the needed steps to reproduce it. Although the db is running in a container, I'm using a volume to store the data locally so we don't lose it if the container goes down.

  @mloviska is now trying to make the JS pg connector work, so we can load the data in JS to be plotted afterwards.

×
Edit Comment 6128
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

* Use asterisks
* for lists

This is an [example link](http://example.com/)

This is an ![example image](http://paste.opensuse.org/view/raw/68957446)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
I managed to have postgres up and running in a container with some security access to push data to it. The problem of postgres is that it doesn't come with rest api by default like for influxdb. Since we don't want to install tools in client side (openQA) like `psql`, we need to have some REST API for postgres. The solution is PostgREST: https://postgrest.org/en/stable/ Which is basically a proxy between the DB and the remote http requests. It offers a simple api to get and push data with simple curl command. In the confluence page I am adding all the needed steps to reproduce it. Although the db is running in a container, I'm using a volume to store the data locally so we don't lose it if the container goes down. @mloviska is now trying to make the JS pg connector work, so we can load the data in JS to be plotted afterwards.
 
×
Reply to jlausuch
I managed to have postgres up and running in a container with some security access to push data to it. The problem of postgres is that it doesn't come with rest api by default like for influxdb. Since we don't want to install tools in client side (openQA) like psql, we need to have some REST API for postgres. The solution is PostgREST: https://postgrest.org/en/stable/ Which is basically a proxy between the DB and the remote http requests. It offers a simple api to get and push data with simple curl command.
In the confluence page I am adding all the needed steps to reproduce it. Although the db is running in a container, I'm using a volume to store the data locally so we don't lose it if the container goes down.
@mloviska is now trying to make the JS pg connector work, so we can load the data in JS to be plotted afterwards.
---
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

* Use asterisks
* for lists

This is an [example link](http://example.com/)

This is an ![example image](http://paste.opensuse.org/view/raw/68957446)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
 
• 

  almost 2 years ago by jlausuch | Reply

  This is the code to push the container image information to the database: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15166

  For the Front-End, we have created a new repository: https://github.com/ilausuch/image-data-ng

×
Edit Comment 6293
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

* Use asterisks
* for lists

This is an [example link](http://example.com/)

This is an ![example image](http://paste.opensuse.org/view/raw/68957446)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview
This is the code to push the container image information to the database: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15166 For the Front-End, we have created a new repository: https://github.com/ilausuch/image-data-ng
 
×
Reply to jlausuch
This is the code to push the container image information to the database: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15166
For the Front-End, we have created a new repository: https://github.com/ilausuch/image-data-ng
---
# A First Level Header
## A Second Level Header

Use one asterisk to *emphasize*

Use two asterisks for **strong emphasis**

* Use asterisks
* for lists

This is an [example link](http://example.com/)

This is an ![example image](http://paste.opensuse.org/view/raw/68957446)

This is a user link @hans

This is a project link hw#some-cool-title
More Complex Markdown Help
Formatting Help
• Edit
• Preview