char/llm-py-web
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

dont clear the system prompt actually

Browse source 
simonw plugins (e.g. llm-anthropic, llm-gemini) don't send the system
prompt as a {role: system} message in build_messages, so the model will
'forget' its system intsructions if it's not retransmitted every time :/
This commit is contained in:
Charlotte Som 2025-03-26 20:55:58 +00:00
parent 09f649ebad
commit 77e7e0b0cf
1 changed files with 3 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
server/inference.py
View file

@ -59,11 +59,12 @@ async def connect_to_conversation(ws: WebSocket):
if conversation_id == "new":
await ws.send_text(json({"i": conversation.id}))
first = True
async for message in ws.iter_text():
if system_prompt:
if first:
await ws.send_text(json({"sys": system_prompt}))
response = conversation.prompt(message, system=system_prompt, stream=True)
system_prompt = None
first = False
response_tid = tid_now()
await ws.send_text(json({"u": message}))