Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sqlflow integrate with zeppelin (SQLFlowInterpreter.java) #11

Closed
wants to merge 5 commits into from
Closed

Conversation

Klay1124
Copy link
Contributor

@Klay1124 Klay1124 commented May 31, 2021

fix #3

This file is to let sqlflow inherit the interpreter interface provided by Zeppelin, so as to connect the two and realize the abstract class of Zeppelin. In this file, you can read the configuration file parameters, construct the sqlflow client, start and close the interpreter, and so on.

Create a new class named org.Apache.Zeppelin.Sqlflow and let it inherit from org.Apache.Zeppelin.Interpreter.Interpreter. Since an abstract class is inherited, constructors and overridden methods are required, with 6 methods:

// Initialize interpreter
public void open();

// Close the interpreter and free resources
public void close();

// Run the interpreter and return the result. 'query' is the code entered in Zeppelin, and 'interpretercontext' is the additional running context information
public InterpreterResult interpret(String query, InterpreterContext interpreterContext);

// Canceling the run will interrupt the execution of the interpret method
public void cancel();

// For processing types of dynamic tables, please refer to: https://zeppelin.apache.org/docs/0.8.2/usage/dynamic_form/intro.html
public ForType getFormType();

// Get the running progress, the return value is 0-100
public int getProgress();

Among these methods, the most critical one is public InterpreterResult interpret(String query, InterpreterContext interpreterContext); It is a bridge between sqlflow and Zeppelin. The collection, creation and release of user information will be realized through this method.

Examples are as follows:

@Override
public InterpreterResult interpret(String query, InterpreterContext context)
		throws InterpreterException {

	if (StringUtils.isBlank(query)) {
		return new InterpreterResult(InterpreterResult.Code.SUCCESS);
	}

	// Create user
	SQLFlow client = EnvironmentSpecificSQLFlowClient.getClient(
			new MessageHandlerZeppelin(context), parameters);
        try {
	        // Run custom sqlflow code
	        client.run(query);
	        // Release user
		client.release();

                return new InterpreterResult(InterpreterResult.Code.SUCCESS);
	} catch (InterruptedException e) {
		LOGGER.error("client error", e);
                return new InterpreterResult(InterpreterResult.Code.ERROR);
	}
}

The specific code can refer to SQLFlowInterpreter.java

private Map<String, String> parameters = new HashMap<String, String>();

public static final String SQLFLOW_SERVERADDR = "sqlflow.serverAddr";
public static final String MYSQL_USERNAME = "mysql.username";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we only support mysql databases?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't just support mysql database. For other SQL engines, including Oracle, Hive, SparkSQL, Flink, etc., developers can expand and improve according to actual needs.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we still use mysql.username to specific the hive username ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This initial version has not yet opened the connection to the hive database.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This initial version has not yet opened the connection to the hive database.

Let's do this under the new issue.

// Create user
SQLFlow client = EnvironmentSpecificSQLFlowClient.getClient(
new MessageHandlerZeppelin(context), parameters);

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

useless blank.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

useless blank removed.

@Override
public InterpreterResult interpret(String query, InterpreterContext context)
throws InterpreterException {

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blank.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

useless blank removed.

return new InterpreterResult(InterpreterResult.Code.ERROR);
}
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blank

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

useless blank removed.

Klay1124 and others added 2 commits June 7, 2021 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Architecture of zeppelin-sqlflow
2 participants