Feature: xorbitsai/inference support

[imotai]

Context

for a better and stable inference performance

Current Solution
the client code

octogen/agent/src/og_agent/codellama_client.py

Lines 37 to 56 in 9fc446b

	prompt = f"""{self.prefix}
	{chat_history}
	{self.user_name}: {user_input}
	{self.ai_name}:"""
	logging.info(f"{prompt}")
	data = {
	"n_predict": 500,
	"grammar": self.grammar,
	"prompt": prompt,
	"temperature": temperature,
	"stream": True,
	"repeat_last_n": 256,
	"top_p": 0.9,
	"stop": [
	"</s>",
	"\n",
	"%s:" % self.ai_name,
	"%s:" % self.user_name,
	],
	}

the llama server

octogen/docker/start_all.sh

Lines 24 to 26 in 9fc446b

	echo "start codellama with model name $3"
	mkdir -p ${ROOT_DIR}/model_server
	cd ${ROOT_DIR}/model_server && hap run -n codellama -- server -m ../model/$3 --alias codellama --host 127.0.0.1 --port 8080