Generate datasource code ?

[tofkamp]

I don't have a lot of knowledge about the inner works of a mikrotik switch, but I have thought of something and wonder if it is possible.
Browsing through your go-code, I noticed that the datasource-interfaces.go is almost a copy of the output of the command: "curl -k -u admin:password https:///rest/interface/ether1".
I am not a go programmer, but I can read it. I can program in python. Would it be an idea we make something like this:

1. Request a structure from a mikrotik switch (REST/curl/whatever).
2. Deduce the schema by using regular expression to determine the type.
3. Put this information in a database.
4. Use a good macro processor to produce the datasource go-files from this information.
5. Do this for a lot of commands eg. /interface/ether1 or /system/resource or /system/clock or ....

Because this is all done in code, it is easy to repeat it on a new firmware version of ROS. If there are schema changes between firmware version, it can be detected.

If the above is possible, is it also possible to generate terraform resource go-code automatically ?
If a human enriches the schema which tells if a field/attribute is mandatory,optional or readonly. Can it be possible to generate go-code for terraform resources ? Or even documentation, if a human provides some descriptive text about a field of structure ?
What do you think about this idea ?

[tofkamp]

no interesting for anyone, so I close it

[vaerh]

I apologize for such a long response!

I use an idea similar to yours in part when adding resources. REST is used to request data (to check if the schema is filled correctly). From Wiki MT, data is extracted that describes the future schema. And a script is used to generate a skeleton that requires further validation. You can take a look #161
The biggest problem is exactly the verification of the newly created schema. Unfortunately there are some wonders described in my comment

@maksym-nazarenko made such a code generator in a parallel project, but I haven't looked at it to see if it's applicable here.

If you have any parts of the finished code I would be interested to see them.

[maksym-nazarenko]

@vaerh our code-generator relies on the client struct definition to generate Terraform resource/datasource - we have 2 spearate Go modules: 1 for API client, and 1 for Terraform provider.
If I am not mistaken, your provider generates MT commands for API without any client code, so I am not sure our approach to generate code fits your needs.

[vaerh]

@maksym-nazarenko

Maksym, thanks for the answer!
How's the progress with the move to framework? I with cloudinit-drive provider was frustrated with a lot of removed functionality.

[maksym-nazarenko]

@vaerh the issue is closed: ddelnano/terraform-provider-mikrotik#145
We migrated all our resources.
We faced issues with default value in documentation generating tool and something minor (can't remember), but it's good enough though.

Maybe, we don't use functionality you mentioned, so it is not visible to us yet 😄

[vaerh]

@maksym-nazarenko
You guys are great! 💪

Of course working with all the data at work is just fine, but some points complicate development: impossibility of DiffSuppress as in SDK, impossibility to change data when saving to state file, very verbose validators, etc.
I will probably not take up this move.

[maksym-nazarenko]

yeah, missing DiffSupress forced us to write custom logic for Update() method of the resource 😬

[vaerh]

That's the secret knowledge you need to remember 😎

Unfortunately some trivial things present in the SDK have to be implemented manually. Let's hang on until Hashi will not give up back support for old providers. I hope it will be not soon.

[tofkamp]

I wanted to contribute to this project, but I can not write go, but I can copy/past it. I have made a python script to retrieve a JSON-sample from a mikrotik switch command, and generate a working datasource-go file. I tested it with "/ip/arp" and "/certificate". These mikrotik command's produce a list of objects. To generate datasource-go-code for singe object/dict samples, I am looking for an example with just one object/dict (eg. ./system/resource), so I can generate those also. If that succeeds, I will make some documentation for every datasource I make/generate.
Can you provide me with an implementation of a single object datasource ? (eg. /system/resource or any other)
If Hashi ever give up support for SDKv2, I can regenerate the modified datasources code pretty fast.
I wanted to add the pythoncode to this post, but github won't let me :-(

[vaerh]

Ok, I need to see if there will be any problems when implementing data sources for system resources and will write a post.

P. S. You can add the pythoncode in your Gist and provide a link or include in the post between pairs of three backquotes ```

[tofkamp]

I put my python code in my own repository https://github.com/tofkamp/Routeros_datasource_generator/tree/main
I also put the two datasource test files in there. Some path's are hardcoded, and also the login, but not a major problem.

[vaerh]

Looks good.
You need to add tests for each data source. Also I feel I may have to fix the data conversion from MT format to TF format. Not everything is fully implemented there, but we'll see right through the testing process.
I also have a question: would you be willing to build a test lab with Go, TF/Open-ToFu and CHR? I would like to see already tested modules when adding pull requests.

[tofkamp]

I have already a linux server and a CHR mikrotik switch running. I needed it to get a sample, in order to generate the code. On my work I have access to real mikrotik switches (routerboard).
I have tested both go files already by hand. I will have a look at the go-file with the test extension. I try if I can generate those also.

terraform {
  required_providers {
    routeros = {
      source = "terraform-routeros/routeros"
    }
  }
}

provider "routeros" {
  #alias    = "bh1-k1-sw1"
  hosturl  = "https://bh1-k1-sw1.home.arpa"
  username = "admin"
  password = "admin"
  ca_certificate = var.ca_certificate_path
  insecure       = false
}

data "routeros_ip_arps" "arps" {}

output "mikrotik_arps" {
    description = "all known arps"
    value = data.routeros_ip_arps.arps.iparps
}



data "routeros_certificates" "certs" {}

output "mikrotik_certificates" {
    description = "all installed certificates"
    value = data.routeros_certificates.certs.certificates
}

[vaerh]

That's great! Are you using any kind of IDE? Or is all debugging done in the console?

[vaerh]

I've tested the ARP records output option and there's something I want to do: since the main data field doesn't make any sense yet, I suggest calling it "data".

data "routeros_ip_arp" "arp_records" {}

output "mikrotik_arp" {
    description = "all known arp"
    value = data.routeros_ip_arp.arp_records.data
}

[tofkamp]

I changed it to 'data'. In the datasource_ip_routes it is not called "data". Which is an inconsistency. It is a choice.
I also added a generated file containing the lines to add to provider.go.
Rewrote the code to use a dict for all variables to be inserted in the templates.
Also generated test-go files. What must I do with them ? How do you test ?

[vaerh]

I only tested ARP.

There are two ways to run it, either with the built-in tester in VSCode/Goland/etc. or with a command in the console.

Which one should I describe? ... but a little later

[tofkamp]

Is this still relevant ?

[vaerh]

And why not?
I have not closed this PR as it has worth. And this tool can be used to achieve the goal.

[vaerh]

I happen to have closed that issue. I'm gonna open it up. Maybe someone will need it.